Oracle Databases on EMC Symmetrix Storage Systems Solutions Guide

Oracle Databases on EMC
Symmetrix Storage Systems
Solutions Guide
Version 1.1

• Generating Restartable Oracle Copies using Symmetrix Storage
• Oracle Remote Replication and Disaster Restart using Symmetrix Storage
• Oracle Data Layout and Performance using Symmetrix Storage

David Waddill

SOLUTIONS GUIDE

Copyright © 2007 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC
CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY
KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY
OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication
requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com
All other trademarks used herein are the property of their respective owners.
Oracle Databases on EMC Symmetrix Storage Systems
1.1
Solutions Guide
Part Number 300-003-505 Rev A02
H2603

ii

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Contents

Preface..............................................................................................................................xv
Chapter 1

Oracle on Open Systems ................................................................................................ 1-1
1.1

Chapter 2

Oracle overview.................................................................................................. 1-2
1.1.1 Oracle system elements ............................................................................ 1-2
1.1.2 Oracle data elements................................................................................. 1-4
1.2
Storage management .......................................................................................... 1-6
1.3
Cloning Oracle objects or environments ............................................................ 1-6
1.4
Backup and recovery .......................................................................................... 1-6
1.5
Oracle Real Application Clusters ....................................................................... 1-7
1.6
Optimizing Oracle layouts on EMC Symmetrix DMX ...................................... 1-8
1.7
EMC and Oracle integration............................................................................... 1-8
1.7.1 Install base ................................................................................................ 1-9
1.7.2 Joint engineering....................................................................................... 1-9
1.7.3 Joint Services Center ................................................................................ 1-9
EMC Foundation Products............................................................................................. 2-1
2.1
2.2
2.3
2.4

EMC Symmetrix DMX ...................................................................................... 2-4
EMC Solutions Enabler base management ........................................................ 2-4
Change Tracker .................................................................................................. 2-6
EMC Symmetrix Remote Data Facility.............................................................. 2-7
2.4.1 SRDF benefits........................................................................................... 2-8
2.4.2 SRDF modes of operation ........................................................................ 2-8
2.4.3 SRDF device and composite groups......................................................... 2-9
2.4.4 SRDF consistency groups......................................................................... 2-9
2.4.5 SRDF terminology.................................................................................. 2-12
2.4.6 SRDF control operations ........................................................................ 2-14
2.4.7 Failover and failback operations............................................................. 2-17
2.4.8 SRDF/A (Asynchronous) operations ...................................................... 2-19
2.4.9 EMC SRDF/Cluster Enabler solutions ................................................... 2-20
2.5
EMC TimeFinder.............................................................................................. 2-21
2.5.1 TimeFinder/Mirror Establish operations ................................................ 2-22
Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

iii

Contents

Chapter 3

2.5.2 TimeFinder split operations ....................................................................2-23
2.5.3 TimeFinder restore operations ................................................................2-24
2.5.4 TimeFinder consistent split .....................................................................2-24
2.5.5 TimeFinder reverse split..........................................................................2-27
2.5.6 TimeFinder/Clone operations..................................................................2-27
2.5.7 TimeFinder/Snap operations ...................................................................2-29
2.6
EMC Storage Resource Management ...............................................................2-31
2.7
EMC ControlCenter ..........................................................................................2-36
2.8
EMC PowerPath................................................................................................2-38
2.9
EMC Replication Manager ...............................................................................2-40
Creating Oracle Database Clones ...................................................................................3-1
3.1
3.2

Overview.............................................................................................................3-2
Comparing recoverable and restartable copies of databases...............................3-2
3.2.1 Recoverable disk copies ............................................................................3-3
3.2.2 Restartable disk copies ..............................................................................3-3
3.3
Copying the database with Oracle shutdown......................................................3-3
3.3.1 Creating Oracle copies using TimeFinder/Mirror .....................................3-4
3.3.2 Creating Oracle copies using TimeFinder/Clone ......................................3-5
3.3.3 Creating Oracle copies using TimeFinder/Snap........................................3-7
3.4
Copying a running database using EMC consistency technology ......................3-8
3.4.1 Creating Oracle copies using TimeFinder/Mirror .....................................3-9
3.4.2 Creating Oracle copies using TimeFinder/Clone ....................................3-10
3.4.3 Creating Oracle copies using TimeFinder/Snap......................................3-12
3.5
Copying the database with Oracle in hot backup mode....................................3-13
3.5.1 Putting the tablespaces or database into hot backup mode .....................3-14
3.5.2 Taking the tablespaces or database out of hot backup mode ..................3-14
3.5.3 Creating Oracle copies using TimeFinder/Mirror ...................................3-15
3.5.4 Creating Oracle copies using TimeFinder/Clone ....................................3-16
3.5.5 Creating Oracle copies using TimeFinder/Snap......................................3-18
3.6
Replicating Oracle using Replication Manager ................................................3-20
3.7
Transitioning disk copies to Oracle database clones.........................................3-22
3.7.1 Host considerations .................................................................................3-22
3.7.2 Enabling a cold database copy ................................................................3-26
3.7.3 Enabling a restartable database copy ......................................................3-27
3.7.4 Enabling a hot backup database copy .....................................................3-27
3.8
Oracle transportable tablespaces.......................................................................3-28
3.8.1 Benefits and uses of transportable tablespaces........................................3-28
3.8.2 Implementation of transportable tablespaces with EMC TimeFinder and
SRDF .................................................................................................................3-29
3.8.3 Transportable tablespace example ..........................................................3-29
3.9
Cross-platform transportable tablespaces .........................................................3-33
3.9.1 Overview .................................................................................................3-33
3.9.2 Implementing cross-platform transportable tablespaces .........................3-34
3.10 Choosing a database cloning methodology.......................................................3-36

iv

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Contents

Chapter 4

Backing Up Oracle Environments.................................................................................. 4-1
4.1

Chapter 5

Comparing recoverable and restartable copies of databases .............................. 4-2
4.1.1 Recoverable disk copies ........................................................................... 4-2
4.1.2 Restartable disk copies ............................................................................. 4-3
4.2
Database organization to facilitate recovery ...................................................... 4-3
4.3
Oracle backup overview ..................................................................................... 4-4
4.3.1 Online (hot) versus offline (cold) backups ............................................... 4-6
4.3.2 Point-in-time and roll-forward recovery backups..................................... 4-7
4.3.3 Comparing partial and entire database backups ....................................... 4-7
4.3.4 Comparing incremental and full database backups .................................. 4-7
4.4
Using EMC replication in the Oracle backup process........................................ 4-8
4.5
Copying the database with Oracle shutdown ..................................................... 4-9
4.5.1 Creating cold Oracle backup copies using TimeFinder/Mirror ................ 4-9
4.5.2 Creating cold Oracle backup copies using TimeFinder/Clone ............... 4-11
4.5.3 Creating cold Oracle backup copies using TimeFinder/Snap................. 4-12
4.6
Copying a running database using EMC consistency technology.................... 4-14
4.6.1 Creating restartable Oracle backup copies using TimeFinder/Mirror .... 4-14
4.6.2 Creating restartable Oracle backup copies using TimeFinder/Clone ..... 4-16
4.6.3 Creating restartable Oracle backup copies using TimeFinder/Snap ....... 4-17
4.7
Copying the database with Oracle in hot backup mode ................................... 4-19
4.7.1 Putting the tablespaces or database into hot backup mode ..................... 4-19
4.7.2 Taking the tablespaces or database out of hot backup mode .................. 4-20
4.7.3 Creating hot Oracle backup copies using TimeFinder/Mirror................ 4-20
4.7.4 Creating “hot” Oracle backup copies using TimeFinder/Clone ............. 4-22
4.7.5 Creating “hot” Oracle backup copies using TimeFinder/Snap ............... 4-24
4.8
Backing up the database copy .......................................................................... 4-26
4.9
Backups using EMC Replication Manager for Oracle backups ....................... 4-26
4.10 Backups using Oracle Recovery Manager (RMAN)........................................ 4-28
4.11 Backups using TimeFinder and Oracle RMAN ............................................... 4-29
Restoring and Recovering Oracle Databases ................................................................. 5-1
5.1

5.2
5.3

5.4
5.5

Oracle recovery types ......................................................................................... 5-2
5.1.1 Crash recovery.......................................................................................... 5-2
5.1.2 Media recovery ......................................................................................... 5-3
5.1.3 Complete recovery.................................................................................... 5-3
5.1.4 Incomplete recovery ................................................................................. 5-4
5.1.5 Restartable database recovery................................................................... 5-4
Oracle recovery overview................................................................................... 5-5
Restoring a backup image using TimeFinder ..................................................... 5-6
5.3.1 Restore using TimeFinder/Mirror............................................................. 5-7
5.3.2 Restore using TimeFinder/Clone.............................................................. 5-9
5.3.3 Restore using TimeFinder/Snap ............................................................. 5-12
Restoring a backup image using Replication Manager .................................... 5-14
Oracle database recovery procedures ............................................................... 5-15

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

v

Contents

5.5.1 Oracle restartable database recovery procedures ....................................5-16
5.5.2 Oracle complete recovery........................................................................5-17
5.5.3 Oracle incomplete recovery ....................................................................5-18
5.6
Database recovery using Oracle RMAN...........................................................5-20
5.7
Oracle Flashback...............................................................................................5-20
5.7.1 Flashback configuration ..........................................................................5-21
5.7.2 Flashback Query......................................................................................5-22
5.7.3 Flashback Version Query........................................................................5-22
5.7.4 Flashback Transaction Query..................................................................5-22
5.7.5 Flashback Table ......................................................................................5-22
5.7.6 Flashback Drop .......................................................................................5-23
5.7.7 Flashback Database.................................................................................5-24
Chapter 6

Understanding Oracle Disaster Restart and Disaster Recovery......................................6-1
6.1

Definitions...........................................................................................................6-2
6.1.1 Dependent-write consistency ....................................................................6-3
6.1.2 Database restart .........................................................................................6-3
6.1.3 Database recovery .....................................................................................6-3
6.1.4 Roll-forward recovery...............................................................................6-4
6.2
Design considerations for disaster restart and disaster recovery.........................6-4
6.2.1 Recovery Point Objective..........................................................................6-4
6.2.2 Recovery Time Objective..........................................................................6-4
6.2.3 Operational complexity.............................................................................6-5
6.2.4 Source server activity................................................................................6-5
6.2.5 Production impact .....................................................................................6-6
6.2.6 Target server activity.................................................................................6-6
6.2.7 Number of copies of data ..........................................................................6-6
6.2.8 Distance for solution .................................................................................6-6
6.2.9 Bandwidth requirements ...........................................................................6-6
6.2.10 Federated consistency ...............................................................................6-7
6.2.11 Testing the solution ...................................................................................6-7
6.2.12 Cost ...........................................................................................................6-7
6.3
Tape-based solutions...........................................................................................6-8
6.3.1 Tape-based disaster recovery ....................................................................6-8
6.3.2 Tape-based disaster restart ........................................................................6-8
6.4
Remote replication challenges ............................................................................6-8
6.4.1 Propagation delay......................................................................................6-9
6.4.2 Bandwidth requirements ...........................................................................6-9
6.4.3 Network infrastructure ............................................................................6-10
6.4.4 Method of instantiation ...........................................................................6-10
6.4.5 Method of reinstantiation ........................................................................6-10
6.4.6 Change rate at the source site..................................................................6-10
6.4.7 Locality of reference ...............................................................................6-11
6.4.8 Expected data loss ...................................................................................6-11
6.4.9 Failback operations .................................................................................6-11

vi

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Contents

6.5
6.6
6.7

Chapter 7

Array-based remote replication ........................................................................ 6-12
Planning for array-based replication................................................................. 6-12
SRDF/S single Symmetrix to single Symmetrix .............................................. 6-14
6.7.1 How to restart in the event of a disaster.................................................. 6-16
6.8
SRDF/S and consistency groups ...................................................................... 6-17
6.8.1 Rolling disaster ....................................................................................... 6-17
6.8.2 Protection against a rolling disaster ........................................................ 6-19
6.8.3 SRDF/S with multiple source Symmetrix systems................................. 6-20
6.9
SRDF/A ............................................................................................................ 6-22
6.9.1 SRDF/A using a single source Symmetrix system ................................. 6-24
6.9.2 SRDF/A multiple source Symmetrix systems ........................................ 6-25
6.9.3 How to restart in the event of a disaster.................................................. 6-26
6.10 SRDF/AR single hop........................................................................................ 6-27
6.10.1 How to restart in the event of a disaster.................................................. 6-28
6.11 SRDF/AR multihop.......................................................................................... 6-29
6.11.1 How to restart in the event of a disaster.................................................. 6-31
6.12 Database log-shipping solutions....................................................................... 6-31
6.12.1 Overview of log shipping ....................................................................... 6-31
6.12.2 Log-shipping considerations................................................................... 6-31
6.12.3 Log shipping and remote standby database ............................................ 6-34
6.12.4 Log shipping and standby database with SRDF ..................................... 6-36
6.12.5 Data Guard.............................................................................................. 6-36
6.13 Running database solutions .............................................................................. 6-43
6.13.1 Overview ................................................................................................ 6-43
6.13.2 Advanced Replication............................................................................. 6-43
6.13.3 Oracle Streams........................................................................................ 6-44
Oracle Database Layouts on EMC Symmetrix DMX .................................................... 7-1
7.1

The performance stack ....................................................................................... 7-2
7.1.1 Importance of I/O avoidance .................................................................... 7-3
7.1.2 Storage-system layer considerations......................................................... 7-3
7.2
Traditional Oracle layout recommendations ...................................................... 7-4
7.2.1 Oracle’s optimal flexible architecture....................................................... 7-4
7.2.2 Oracle layouts and replication considerations .......................................... 7-5
7.2.3 Automated Storage Management ............................................................. 7-5
7.3
Symmetrix DMX performance guidelines ......................................................... 7-5
7.3.1 Front-end connectivity.............................................................................. 7-6
7.3.2 Symmetrix cache ...................................................................................... 7-7
7.3.3 Back-end considerations......................................................................... 7-14
7.3.4 Additional layout considerations ............................................................ 7-15
7.3.5 Configuration recommendations ............................................................ 7-16
7.4
RAID considerations ........................................................................................ 7-16
7.4.1 Types of RAID ....................................................................................... 7-17
7.4.2 RAID recommendations ......................................................................... 7-20
7.4.3 Symmetrix metavolumes ........................................................................ 7-21

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

vii

Contents

7.5

7.6

7.7

7.8
7.9
Chapter 8

Host- versus array-based striping......................................................................7-21
7.5.1 Host-based striping..................................................................................7-22
7.5.2 Symmetrix-based striping (metavolumes)...............................................7-23
7.5.3 Striping recommendations.......................................................................7-23
Data placement considerations..........................................................................7-24
7.6.1 Disk performance considerations ............................................................7-24
7.6.2 Hypervolume contention.........................................................................7-26
7.6.3 Maximizing data spread across the back end ..........................................7-27
7.6.4 Minimizing disk head movement............................................................7-28
Other layout considerations ..............................................................................7-28
7.7.1 Database layout considerations with SRDF/S.........................................7-28
7.7.2 Database cloning, TimeFinder, and sharing spindles..............................7-29
7.7.3 Database clones using TimeFinder/Snap ................................................7-30
Oracle database-specific configuration settings................................................7-30
The database layout process..............................................................................7-33
7.9.1 Database layout process ..........................................................................7-33

Data Protection ...............................................................................................................8-1
8.1

EMC Double Checksum overview .....................................................................8-2
8.1.1 Traditional methods of preventing data corruption...................................8-2
8.1.2 Data corruption between host and conventional storage...........................8-3
8.1.3 Benefits of checking within Symmetrix....................................................8-3
8.2
Implementing EMC Double Checksum for Oracle.............................................8-3
8.2.1 Other checksum operations .......................................................................8-4
8.2.2 Enabling checksum options.......................................................................8-4
8.2.3 Verifying checksum is enabled .................................................................8-5
8.2.4 Validating for checksum operations..........................................................8-5
8.2.5 Disabling checksum ..................................................................................8-6
8.3
Implementing Generic SafeWrite for generic applications.................................8-6
8.3.1 Torn pages: Using Generic SafeWrite to protect applications ..................8-6
8.3.2 Why generic? ............................................................................................8-7
8.3.3 Where to enable Generic SafeWrite..........................................................8-7
8.3.4 Configuring Generic SafeWrite ................................................................8-8
8.3.5 How to disable Generic SafeWrite..........................................................8-10
8.3.6 Listing Generic SafeWrite devices..........................................................8-10
8.3.7 Performance considerations ....................................................................8-11
8.4
Syntax and examples.........................................................................................8-12
Appendix A Related Documents........................................................................................................A-1
A.1

Related documents .............................................................................................A-2

Appendix B Sample SYMCLI Group Creation Commands .............................................................. B-1
B.1

Sample SYMCLI group creation commands ..................................................... B-2

Appendix C Related Host Operations ................................................................................................ C-1
C.1

viii

Overview............................................................................................................ C-2
C.1.1 BIN file configuration ..............................................................................C-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Contents

C.1.2 SAN considerations ................................................................................. C-2
C.1.3 Final configuration considerations for enabling LUN
presentation to hosts .............................................................................................. C-3
C.2
Presenting database copies to a different host ....................................................C-4
C.2.1 AIX considerations .................................................................................. C-4
C.2.2 HP-UX considerations ............................................................................. C-6
C.2.3 Linux considerations................................................................................ C-9
C.2.4 Solaris considerations ............................................................................ C-10
C.2.5 Windows considerations........................................................................ C-12
C.2.6 Windows Dynamic Disks ...................................................................... C-14
C.3
Presenting database copies to the same host ....................................................C-14
C.3.1 AIX considerations ................................................................................ C-15
C.3.2 HP-UX considerations ........................................................................... C-15
C.3.3 Linux considerations.............................................................................. C-18
C.3.4 Solaris considerations ............................................................................ C-19
C.3.5 Windows considerations........................................................................ C-19
Appendix D Sample Database Cloning Scripts ................................................................................. D-1
D.1

Sample script to replicate a database................................................................. D-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

ix

Contents

x

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Figures

Figures

Figure 1-1 Oracle instance and database ................................................................... 1-3
Figure 1-2 Physical data elements in an Oracle configuration .................................. 1-4
Figure 1-3 Relationship between data blocks, extents, and segments ....................... 1-6
Figure 1-4 Oracle two-node RAC configuration ....................................................... 1-8
Figure 2-1 Basic synchronous SRDF configuration ................................................... 2-8
Figure 2-2 SRDF consistency group......................................................................... 2-11
Figure 2-3 SRDF establish and restore control operations ....................................... 2-15
Figure 2-4 SRDF failover and failback control operations....................................... 2-17
Figure 2-5 Geographically distributed four-node EMC SRDF/CE clusters ............. 2-21
Figure 2-6 EMC Symmetrix configured with standard volumes and BCVs ............ 2-22
Figure 2-7 ECA consistent split across multiple database associated hosts ............. 2-25
Figure 2-8 ECA consistent split on a local Symmetrix............................................. 2-26
Figure 2-9 Creating a copy session using the symclone command .......................... 2-28
Figure 2-10 TimeFinder/Snap copy of a standard device to a VDEV ...................... 2-30
Figure 2-11 SRM commands .................................................................................... 2-32
Figure 2-12 ControlCenter family overview............................................................. 2-36
Figure 3-1 Copying a cold (shutdown) Oracle database with TimeFinder/Mirror ..... 3-4
Figure 3-2 Copying a cold Oracle database with TimeFinder/Clone ......................... 3-6
Figure 3-3 Copying a cold Oracle database with TimeFinder/Snap........................... 3-7
Figure 3-4 Copying a running Oracle database with TimeFinder/Mirror................... 3-9
Figure 3-5 Copying a running Oracle database with TimeFinder/Clone.................. 3-11
Figure 3-6 Copying a running Oracle database with TimeFinder/Snap ................... 3-12
Figure 3-7 Copying an Oracle database in hot backup mode
with TimeFinder/Mirror ............................................................................................... 3-15
Figure 3-8 Copying an Oracle database in hot backup mode
with TimeFinder/Clone ................................................................................................ 3-17
Figure 3-9 Copying an Oracle database in hot backup mode
with TimeFinder/Snap.................................................................................................. 3-19
Figure 3-10 Using Replication Manager to make a TimeFinder copy of Oracle...... 3-21
Figure 4-1 Database organization to facilitate recovery ............................................. 4-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

xi

Figures

Figure 4-2 Copying a cold Oracle database with TimeFinder/Mirror.......................4-10
Figure 4-3 Copying a cold Oracle database with TimeFinder/Clone........................4-11
Figure 4-4 Copying a cold Oracle database with TimeFinder/Snap .........................4-13
Figure 4-5 Copying a running Oracle database with TimeFinder/Mirror .................4-15
Figure 4-6 Copying a running Oracle database using TimeFinder/Clone.................4-16
Figure 4-7 Copying a running Oracle database with TimeFinder/Snap....................4-18
Figure 4-8 Copying an Oracle database in hot backup mode
with TimeFinder/Mirror................................................................................................4-21
Figure 4-9 Copying an Oracle database in hot backup mode
with TimeFinder/Clone.................................................................................................4-23
Figure 4-10 Copying an Oracle database in hot backup mode
with TimeFinder/Snap ..................................................................................................4-25
Figure 4-11 Using RM to make a TimeFinder copy of Oracle .................................4-27
Figure 5-1 Restoring a TimeFinder copy, all components ..........................................5-7
Figure 5-2 Restoring a TimeFinder copy, data components only ...............................5-8
Figure 5-3 Restoring a TimeFinder/Clone copy, all components .............................5-10
Figure 5-4 Restoring a TimeFinder/Clone copy, data components only...................5-10
Figure 5-5 Restoring a TimeFinder/Snap copy, all components ...............................5-12
Figure 5-6 Restoring a TimeFinder/Snap copy, data components only ....................5-13
Figure 5-7 Restoring Oracle using EMC Replication Manager ................................5-15
Figure 6-1 Database components for Oracle .............................................................6-13
Figure 6-2 Synchronous replication internals............................................................6-15
Figure 6-3 Rolling disaster with multiple production Symmetrix systems ...............6-18
Figure 6-4 Rolling disaster with SRDF consistency group protection......................6-19
Figure 6-5 SRDF/S with multiple source Symmetrix systems and ConGroup
protection .................................................................................................................6-21
Figure 6-6 SRDF/A replication internals ..................................................................6-23
Figure 6-7 SRDF/AR single-hop replication internals..............................................6-27
Figure 6-8 SRDF/AR multihop replication Internals ................................................6-29
Figure 6-9 Log shipping and remote standby database .............................................6-34
Figure 6-10 Sample Oracle10g Data Guard configuration........................................6-39
Figure 6-11 “No data loss” standby database............................................................6-42
Figure 7-1 The performance stack...............................................................................7-3
Figure 7-2 Relationship between host block size and IOPS/throughput .....................7-7
Figure 7-3 Performance Manager graph of write-pending limit for a single
hypervolume .................................................................................................................7-12
Figure 7-4 Performance Manager graph of write-pending limit for a four-member
metavolume .................................................................................................................7-13
Figure 7-5 Write workload for a single hyper and a striped metavolume .................7-14
Figure 7-6 3+1 RAID 5 layout detail ........................................................................7-17
Figure 7-7 Anatomy of a RAID 5 random write .......................................................7-18
Figure 7-8 Optimizing performance with RAID 5 sequential writes ........................7-19
Figure 7-9 Disk performance factors.........................................................................7-26
Figure 8-1 Synchronous replication internals............................................................8-11
Figure C-1 Windows Disk Management console..................................................... C-13

xii

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Tables

Tables

Table 1-1
Table 2-1
Table 2-2
Table 2-3
Table 2-4
Table 2-5
Table 2-6
Table 2-7
Table 3-1
Table 3-2
Table 6-1
Table 7-1
Table 8-1

Oracle processes ......................................................................................... 1-3
SYMCLI base commands........................................................................... 2-5
TimeFinder device type summary ............................................................ 2-29
Data object SRM commands .................................................................... 2-34
Data object mapping commands .............................................................. 2-34
File system SRM command...................................................................... 2-35
File system SRM commands .................................................................... 2-35
SRM statistics command .......................................................................... 2-36
Comparison of database cloning technologies ......................................... 3-36
Database cloning requirements and solutions .......................................... 3-36
Background processes for managing a Data Guard environment ............ 6-39
Initialization parameters ........................................................................... 7-32
Background processes for managing a Data Guard environment ............ 8-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

xiii

Tables

xiv

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Preface

This document describes how the EMC Symmetrix system manages Oracle databases on
UNIX and Windows. Additionally, this document provides a general description of the
Oracle RDBMS and EMC products and utilities that can be used for Oracle
administration. EMC Symmetrix storage systems and EMC software products and
utilities are used to clone Oracle environments and to enhance database and storage
management backup and recovery procedures. Using EMC products and utilities to
manage Oracle environments reduces the following:
♦ Database and storage management administration
♦ CPU resource consumption
♦ The time required to clone or recover Oracle systems
As part of an effort to improve and enhance the performance and capabilities of its product line,
EMC from time to time releases revisions of its hardware and software. Therefore, some
functions described in this guide may not be supported by all revisions of the software or
hardware currently in use. For the most up-to-date information on product features, refer to your
product release notes.
Audience

The intended audience is systems administrators, Oracle database administrators, and
storage management personnel responsible for managing Oracle databases on opensystems platforms. The information in this document is based on Oracle10g. In this
document, open-systems platforms are UNIX operating systems (including AIX, HPUX, Linux, and Solaris), as well as Microsoft Windows platforms.
Organization

The document is divided into the following chapters:
Chapter 1, Oracle on Open Systems, provides a high-level overview of Oracle.
Chapter 2, EMC Foundation Products, describes EMC products used to support the
management of Oracle environments.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

xv

Preface

Chapter 3, Creating Oracle Database Clones, describes procedures to clone Oracle
instances. It also discusses procedures to clone Oracle objects within and across Oracle
instances using Oracle Transportable Tablespaces and EMC TimeFinder.
Chapter 4, Backing Up Oracle Environments, describes how to back up Oracle
environments and objects with Oracle Recovery Manager and EMC products including
TimeFinder and SRDF.
Chapter 5, Restoring and Recovering Oracle Databases, describes how to recover Oracle
environments and objects, based upon the type of backups that were previously
performed.
Chapter 6, Understanding Oracle Disaster Restart and Disaster Recovery, describes the
difference between using traditional recovery techniques versus EMC restart solutions.
Chapter 7, Oracle Database Layouts on EMC Symmetrix DMX, describes Oracle
RDBMS on EMC Symmetrix DMX data layout recommendations and best practices.
The appendixes provide sample code, which supplement procedures described in the
document.
Appendix A lists additional reference documents related to information found in this
guide.
Appendix B gives example commands for Solutions Enabler.
Appendix C provides information on host-related activities for cloning, backup, and
restore.
Appendix D has a sample script for cloning Oracle databases.
The references section lists documents that contain more information on these topics.
Examples provided in this document cover methods for performing various Oracle
functions using Symmetrix systems with EMC software. These examples were
developed for laboratory testing and may need tailoring to suit other operational
environments. Any procedures outlined in this document should be thoroughly tested
prior to production implementation.

xvi

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 1

Oracle on Open
Systems

This chapter presents these topics:
1.1
1.2
1.3
1.4
1.5
1.6
1.7

Oracle overview.................................................................................................. 1-2
Storage management .......................................................................................... 1-6
Cloning Oracle objects or environments ............................................................ 1-6
Backup and recovery .......................................................................................... 1-6
Oracle Real Application Clusters ....................................................................... 1-7
Optimizing Oracle layouts on EMC Symmetrix DMX ...................................... 1-8
EMC and Oracle integration............................................................................... 1-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

1-1

Oracle on Open Systems

The Oracle RDBMS on open systems first became available in 1979 and has steadily
grown to become the market share leader in enterprise database solutions. With a wide
variety of features and functionality, Oracle provides a stable platform for handling
concurrent, read-consistent access to a customer’s application data.
Oracle10g, the latest release of the Oracle RDBMS, comes with a variety of new and
enhanced features over previous versions of the database. Among these are:
♦ Increased self-management through features such as Automatic Undo
Management, Oracle managed files, and mean time to recovery enhancements.
♦ Improved toolsets and utilities such as Recovery Manager (RMAN) and Oracle
Enterprise Manager (OEM).
♦ Introduction of Automatic Storage Management (ASM).
♦ Introduction of Database Resource Manager.
♦ Enhancements to Oracle Flashback capabilities.
Oracle’s architectural robustness, scalability, and availability functions have positioned
it as a cornerstone in many customers’ enterprise system infrastructures. A large number
of EMC customers use Oracle in open-systems environments to support large, missioncritical business applications.

1.1 Oracle overview
The Oracle RDBMS can be configured in multiple ways. The requirement for 24x7
operations, replication and disaster recovery, and the capacity of the host(s) that will
contain the Oracle instance(s) will, in part, determine how the Oracle environment must
be architected.

1.1.1 Oracle system elements
An Oracle database consists of three basic components: memory structures, processes,
and datafiles. An Oracle instance is defined as the System Global Area and the
associated background processes. Figure 1-1 on page 1-3 shows a simplified example of
the Oracle components.

1-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle on Open Systems

Figure 1-1

Oracle instance and database

In this example, the System Global Area (SGA) contains the basic memory structures
the Oracle database requires to function. The SGA contains memory structures for the
DBMS (data dictionary), data (database block buffers), logs (redo log buffers), SQL
(shared pool), and control information for server processes (program global area).
In addition to the SGA, the Oracle instance contains background processes. The
processes are started when the instance is initiated; they enable Oracle to perform tasks
such as reading and writing between the data files and the SGA, managing I/O to the
redo logs, performing archiving between the redo and archive logs, and connecting users
to the database. Table 1-1 on page 1-3 describes the Oracle processes shown in Figure
1-1.
Table 1-1

Oracle processes
Process

Description

DBWn (Database
Writer)

Writes data from the database block buffers to the datafiles on disk. Up to
20 database writer processes can be started per Oracle instance.

LGWR (Log
Writer)

Manages the redo log buffers and transmitting data from the buffers to the
redo logs on disk.

ARCn (Database
Archiver)

Copies the redo logs to one or more log directories when a log switch
occurs. The ARCn process is only turned on if the database is in
ARCHIVELOG mode. Up to 10 archive processes can be started per
Oracle instance.

CKPT
(Checkpoint)

Enhances performance during a system checkpoint. This process is
optional. When the Oracle system checkpoints, DBWn needs to commit
data to disk and update the datafile headers. When started, the CKPT
process offloads updating the headers from the DBWn.

Oracle overview

1-3

Oracle on Open Systems

SMON (System
Monitor)

Performs recovery at instance startup. It coalesces free extents in the
datafiles and cleaning up temporary segments of failed user processes.

PMON (Process
Monitor)

Cleans up after a user process fails. The process frees up resources
including database locks and the blocks in the buffer cache of the failed
process.

Snnn (Server
processes)

Connect user processes to the database instance. Server processes can
either be dedicated or shared, depending on user requirements and the
amount of host memory available.

Additional database processes may be started depending on the system configuration.
Some of these processes include RECO, QMNn, Jnnn, and MMON.
Finally, the datafiles are the physical structures that store data on disk. These files can
be created within a file system or as raw partitions. Oracle uses data files to create the
logical structures within the database that hold user information. These logical
structures include tablespaces, segments, extents, and data blocks.

1.1.2 Oracle data elements
Oracle maintains a set of database elements critical to the operation of the Oracle
subsystem. These database elements consist of both physical and logical data elements.
1.1.2.1 Physical data elements

The required physical data elements include datafile(s) for the Oracle SYSTEM
tablespace, the control files, the redo logs, and other miscellaneous database files (the
parameter file, alert and trace logs, backup files, and so on). Other physical elements
such as the archive logs and additional tablespaces for data are also typically configured.
A minimal configuration is shown in Figure 1-2 on page 1-4, followed by a description
of each data structure.

Figure 1-2

1-4

Physical data elements in an Oracle configuration

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle on Open Systems

The Oracle SYSTEM tablespace consists of tables of data describing everything defined
to the Oracle instance, including users, tablespaces, tables, indexes, performance
information, and so on. This tablespace is the only one required, although in practice,
other tablespaces containing data are typically created.
The Oracle control files consist of one or more configuration files (the control file is
typically multiplexed onto separate physical spindles) that contain the name of the
database, the name and location of all database datafiles and redo logs, redo and archive
log history information, checkpoint information, and other information needed at system
startup and while the database is running.
Oracle redo logs contain data changes, pre- and postdata images, and other significant
events. All changes to the database are written to the redo logs unless logging is
explicitly turned off. Two or more redo logs are configured, and normally the logs are
multiplexed to prevent data loss in the event that database recovery is required.
Archive logs are offloaded copies of the redo logs and are normally required for
recovering an Oracle database. Archive logs can be multiplexed, both locally and
remotely.
Oracle binaries are the executables and libraries used to initiate the Oracle instance.
Along with the binaries, Oracle uses many other datafiles to manage and monitor the
database. These files include the initialization parameter file (init<sid>.ora), the server
parameter file (SPFILE), the alert log, and trace files.
1.1.2.2 Logical data elements

Datafiles are the primary physical data element. Oracle tablespaces are the logical
element configured on top of thedatafiles. Oracle tablespaces are used as containers to
hold the customer’s information. Each tablespace is built on one or more of thedatafiles.
Tablespaces are the containers for the underlying Oracle logical data elements. These
logical elements include data blocks, extents, and segments. Data blocks are the
smallest logical elements configurable at the database level. Data blocks are grouped
into extents that are then allocated to segments. Types of segments include table, table
partition, index, index partition, cluster, rollback, deferred rollback, cache, lobsegment,
lobindex, and temporary segments.
Figure 1-3 on page 1-6 shows the relationship between the data blocks, extents, and
segments.

Oracle overview

1-5

Oracle on Open Systems

Figure 1-3

Relationship between data blocks, extents, and segments

1.2 Storage management
Standard Oracle backup, recovery, and cloning methods can be difficult to manage and
time-consuming. EMC provides alternative solutions to traditional Oracle utilities for
cloning systems, backup and recovery, and disaster recovery or business continuance.

1.3 Cloning Oracle objects or environments
EMC technology enables creation of an instant point-in-time copy of an Oracle database
system. The cloned copy is an identical environment to its source, and can be used for
other processing purposes such as backup, recovery, offline reporting, and testing.
Transportable tablespaces are an alternative to cloning an entire Oracle database.
Through Oracle’s transportable tablespaces, it is possible to clone an individual
tablespace and present it to a different Oracle database environment. Clone creation is
facilitated through the use of EMC products such as TimeFinder®.
In addition to transportable tablespaces, Oracle also may clone or replicate individual
database objects, such as tables, in a variety of ways. Methods include trigger-based
mechanisms such as snapshots, Oracle’s Advanced Replication, and Oracle Data Guard.

1.4 Backup and recovery
Backup and recovery operations using Oracle utilities typically require intervention by
experienced personnel and can be both labor- and resource-intensive in large Oracle
environments. Recovery of large Oracle instances, such as in SAP or PeopleSoft

1-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle on Open Systems

environments, are especially complex because the entire system is basically a large
referential set. All data in the set, including the database and associated application
files, needs to be recovered together and to the same recovery point.
Dynamic manipulation of objects and application-maintained referential integrity further
complicates recovery efforts. Traditional Oracle recovery techniques require multiple
passes of the data, which can greatly impact recovery times. Such techniques are
generally unworkable in large Oracle environments due to the time required to recover
all objects. EMC hardware and software are used to make the process faster and more
effective.
In addition to traditional backup and recovery operations, Oracle provides the Recovery
Manager (RMAN) utility. RMAN provides a wide range of backup and recovery
procedures through either a command line interface on a client host or a GUI interface in
Enterprise Manager. RMAN performs backup or recovery operations by integrating
with sessions running on the target database host. Remote procedure calls (RPCs) to
specialized packages stored in the target database are then made that execute in the
backup or recovery of the database. RMANalso may be configured as a repository for
historical backup information that supplements records written by the utility into the
database control file.

1.5 Oracle Real Application Clusters
Typically, Oracle is configured with a single instance that attaches to a single database.
However, Oracle can be configured with multiple host instances connecting to a single
database. This configuration, which originally was called Oracle Parallel Server (OPS)
in Oracle versions prior to release Oracle9i, is now known as Oracle Real Application
Clusters (RAC). Implementations of Oracle RAC are configured to enhance
performance, scalability, and availability over a stand-alone Oracle database.
An Oracle RAC environment consists of multiple Oracle instances running in separate
host environments that share access to a single Oracle database. Each instance contains
its own memory structures and processes. In addition, each instance contains its own set
of redo logs and undo segments. Each instance shares access to the datafiles making up
the database. Since all hosts must have access to all database datafiles, concurrent
access to the data files through the use of cluster manager is required. This also permits
one host to assume control of all datafiles in the event of an instance failure requiring
recovery.
Performance and scalability are enhanced in an Oracle environment because host-based
resource limitations such as CPU and memory constraints are overcome by permitting
two or more host instances to attach to the same database. For example, in a
homogeneous host environment, near-linear scaling of host resources is achieved by
employing Oracle RAC. Additionally, because multiple hosts are configured with
access to the database, availability is increased. In the event of a failure to one host or
database instance, user connections are failed over to the surviving cluster members
ensuring continuous operations.
Figure 1-4 on page 1-8 shows a typical Oracle RAC configuration with two member
nodes. Each member of the group has its own SGA, redo logs, and undo space. Though
not shown here, each member also has its own set of initialization and parameter files.

Oracle Real Application Clusters

1-7

Oracle on Open Systems

Concurrent access to each data file is managed through the cluster management
software. Locking and interinstance management is communicated through a network
interconnect between the RAC nodes.

Figure 1-4

Oracle two-node RAC configuration

1.6 Optimizing Oracle layouts on EMC Symmetrix DMX
A primary concern for DBAs and system administrators when configuring an Oracle
databases on an EMC® Symmetrix DMX™ is the appropriate data layout on the storage.
Maximizing performance, availability, and recoverability of the database requires a
thorough understanding of the I/O characteristics, uptime requirements, backup, and
cloning needs. Careful consideration and planning of the back-end configuration,
including RAID, physical spindles, number of front-end directors and HBAs, as well as
layout of the database on the back-end of the Symmetrix® system is necessary. These
considerations ensure the database implementation successfully meets all business
requirements.

1.7 EMC and Oracle integration
The EMC/Oracle partnership was established in 1995 and continues to the present.
Through joint engineering efforts, certification testing, collaborative solution offerings,
and the Joint Services Center, EMC and Oracle maintain strong ties to ensure successful
product integration for customers’ mission-critical database systems. Efforts such as
Project MegaGrid, a collaboration between EMC, Oracle, Dell, and Intel to design,
create, and document Enterprise Grid Computing best-practice infrastructures, help the
two companies to provide customers with scalable, high-performance, highly available,
and cost-effective database solutions.

1-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle on Open Systems

1.7.1 Install base
With more than 40,000 mutual customers, EMC and Oracle are recognized as the leaders
in automated networked storage and enterprise software respectively. The EMC
Symmetrix DMX offers the highest levels of performance and availability along with
industry-leading software for successfully managing and maintaining complex Oracle
database environments.

1.7.2 Joint engineering
Engineers for EMC and Oracle continue to work together to develop integrated
solutions, document best practices, and ensure interoperability for customers deploying
Oracle databases in EMC Symmetrix DMX storage environments. Key EMC
technologies such as TimeFinder®, SRDF®, and Double Checksum are certified through
Oracle’s Storage Certification Program (OSCP). Engineering efforts continue between
the two companies to ensure successful integration between each company’s products.

1.7.3 Joint Services Center
EMC and Oracle maintain a Joint Services Center to handle specific customer questions
and issues relating to the database in EMC Symmetrix DMX environments. When level
1 tech support from either company requires assistance with joint EMC/Oracle-related
issues, calls are automatically escalated to this service center. Based in Hopkinton, MA
this Joint Service Center provides answers to EMC- and Oracle-related questions from
leading support specialists trained in both database and storage platforms.

EMC and Oracle integration

1-9

Oracle on Open Systems

1-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 2

EMC Foundation
Products

This chapter presents these topics:
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9

EMC Symmetrix DMX ...................................................................................... 2-4
EMC Solutions Enabler base management ........................................................ 2-4
Change Tracker .................................................................................................. 2-6
EMC Symmetrix Remote Data Facility.............................................................. 2-7
EMC TimeFinder.............................................................................................. 2-21
EMC Storage Resource Management............................................................... 2-31
EMC ControlCenter.......................................................................................... 2-36
EMC PowerPath ............................................................................................... 2-38
EMC Replication Manager............................................................................... 2-40

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

2-1

EMC Foundation Products

EMC provides many hardware and software products that support application
environments on Symmetrix arrays. The following products, which are highlighted and
discussed, were used and/or tested with the Oracle products discussed in this document.
This chapter provides a technical overview of the EMC products used in this solution
guide.
♦ EMC Symmetrix — EMC offers an extensive product line of high-end storage
solutions targeted to meet the requirements of customers’ mission-critical databases
and applications. The Symmetrix product line includes the DMX Direct Matrix
Architecture™ series, and the 8000, 5000, and 3000 series family. EMC Symmetrix
is a fully redundant, high-availability storage processor, providing non-disruptive
component replacements and code upgrades. The Symmetrix system features high
levels of performance, data integrity, reliability, and availability.
♦ EMC Solutions Enabler — Solutions Enabler is a package that contains the
SYMAPI runtime libraries and the SYMCLI command line interface. SYMAPI
provides the interface to the Symmetrix operating system. SYMCLI are set of
commands that can be invoked via the command line or within scripts. These
commands can be used to monitor device configuration and status, and to perform
control operations on devices and data objects within a storage complex. The target
storage environments are typically Symmetrix based, however, CLARiiON® arrays
can also be managed via the SYMCLI SRM component.
♦ EMC Symmetrix Remote Data Facility (SRDF) — SRDF is a business continuity
software solution that replicates and maintains a mirror image of data at the storage
block level in a remote Symmetrix system. The SRDF component extends the basic
SYMCLI command set of Solutions Enabler to include commands that specifically
manage SRDF.
•

EMC SRDF consistency groups — An SRDF consistency group is a collection
of related Symmetrix devices that are configured to act in unison to maintain
data integrity. The devices in consistency groups can be spread across multiple
Symmetrix systems.

♦ EMC TimeFinder — TimeFinder is a family of products that enable LUN-based
replication within a single Symmetrix array. Data is copied from Symmetrix devices
using array-based resources without using host CPU or I/O. The source Symmetrix
devices remain online for regular I/O operations while the copies are created. The
TimeFinder family has three separate and distinct software products,
TimeFinder/Mirror, TimeFinder/Clone, and TimeFinder/Snap:

2-2

•

TimeFinder/Mirror allows users to configure special devices, called business
continuance volumes (BCVs), to create a mirror image of Symmetrix standard
devices. Using BCVs, TimeFinder creates a point-in-time copy of data that can
be repurposed. The TimeFinder/Mirror component extends the basic SYMCLI
command set of Solutions Enabler to include commands that specifically
manage Symmetrix BCVs and standard devices.

•

TimeFinder/Clone allows users to make copies of data simultaneously on
multiple target devices from a single source device. The data is available to a
target’s host immediately upon activation, even if the copy process has not
completed. Data may be copied from a single source device to as many as 16
target devices. A source device can be either a Symmetrix standard device or a

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

TimeFinder BCV device. A target device can be any volume in the array of
equal or greater size than the source device with which it is paired.
•

TimeFinder/Snap allows users to configure special devices in the Symmetrix
DMX array called virtual devices (VDEVs) and save area devices (SAVDEVs).
These devices can be used to make pointer-based, space-saving copies of data
simultaneously on multiple target devices from a single source device. The data
is available to a target’s host immediately upon creation. Data may be copied
from a single source device to as many as 15 VDEVs. A source device can be
either a Symmetrix standard device or a TimeFinder BCV device. A target
device is a VDEV. A SAVDEV is a special device without a host address that is
used to hold the changing contents of the source or target device.

♦ EMC Open Replicator — Open Replicator provides block-level data replication and
data migration between a Symmetrix DMX and secondary heterogeneous storage
environments over a SAN. Open Replicator is array-based replication software that
runs exclusively on a Symmetrix DMX that enables copy operations independent of
the host, operating system, and data type.
♦ EMC Change Tracker — EMC Symmetrix Change Tracker software measures
changes to data on a Symmetrix volume or group of volumes. Change Tracker
software is often used as a planning tool in the analysis and design of configurations
that use the EMC TimeFinder or SRDF components to store data at remote sites.
♦ Solutions Enabler Storage Resource Management (SRM) Component — The SRM
component extends the basic SYMCLI command set of Solutions Enabler to include
commands that allow users to systematically find and examine attributes of various
objects on the host, within a specified relational database, or in the EMC enterprise
storage. The SRM commands provide mapping support for relational databases, file
systems, logical volumes and volume groups, as well as performance statistics.
♦ EMC ControlCenter™ — EMC ControlCenter is an integrated family of software
products that enables users to discover, monitor, automate, provision, and report on
storage area networks, host resources, and storage resources across the entire
information environment.
♦ EMC PowerPath® — PowerPath is host-based software that provides I/O path
management. PowerPath operates with several storage systems, on several
enterprise operating systems and provides failover and load balancing transparent to
the host application and database.
♦ EMC SRDF/Cluster Enabler — SRDF/Cluster Enabler (SRDF/CE) for MSCS is one
in a family of automated business restart solutions that integrates EMC SRDF with
cluster technology. SRDF/CE for MSCS provides disaster recovery protection in
geographically distributed clusters.
♦ Connectrix® — Connectrix is a Fibre Channel director or switch that moves
information throughout the SAN environment, enabling the networked storage
solution.
♦ EMC Replication Manager — EMC Replication Manager is software that creates
replicas of mission-critical databases on disk arrays with traditional tape media.
Replication Manager can create a disk replica of data simply, quickly, and
automatically. It automates all tasks and/or procedures related to data replication,

EMC and Oracle integration

2-3

EMC Foundation Products

and reduces the amount of time, resources, and expertise involved with integrating
and managing disk-based replication technologies

2.1 EMC Symmetrix DMX
All Symmetrix systems provide advanced data replication capabilities, full mainframe
and open systems support, and flexible connectivity options, including Fibre Channel,
FICON, ESCON, Gigabit Ethernet, and iSCSI.
Interoperability between Symmetrix storage systems enables current customers to
migrate their storage solutions from one generation to the next, protecting their
investment even as their storage demands expand.
Symmetrix DMX enhanced cache director technology allows configuration of up to 256
GB of cache. The cache can be logically divided into 32 independent regions providing
up to 32 concurrent 500 MB/s transaction throughput.
The Symmetrix on-board data integrity features include:
♦ Continuous cache and on-disk data integrity checking and error detection/correction.
♦ Fault isolation.
♦ Non-disruptive hardware and software upgrades.
♦ Automatic diagnostics and phone-home capabilities.
In addition to the models listed previously, for environments that require ultra-high
performance, EMC provides DMX1000-P and DMX2000-P systems. These two storage
systems are built for extra speed to operate in extreme performance-intensive
environments such as decision support, data warehousing, and other high-volume, backend sequential processing applications.
At the software level, advanced integrity features ensure information is always protected
and available. By choosing a mix of RAID 1 (mirroring), RAID 1/0, and high
performance RAID 5 (3+1 and 7+1) protection, users have the flexibility to choose the
protection level most appropriate to the value and performance requirements of their
information. The Symmetrix DMX is EMC’s latest generation of high-end storage
solutions.
From the perspective of the host operating system, a Symmetrix system appears to be
multiple physical devices connected through one or more I/O controllers. The host
operating system addresses each of these devices using a physical device name. Each
physical device includes attributes, vendor ID, product ID, revision level, and serial ID.
The host physical device maps to a Symmetrix device. In turn, the Symmetrix device is
a virtual representation of a section of the physical disk called a hypervolume.

2.2 EMC Solutions Enabler base management
The EMC Solutions Enabler kit contains all the base management software that provides
a host with SYMAPI-shared libraries and the basic Symmetrix command line interface
(SYMCLI). Other optional subcomponents in the Solutions Enabler (SYMCLI) series
2-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

enable users to extend functionality of the Symmetrix systems. Three principle subcomponents are:
♦ Solutions Enabler SYMCLI SRDF, SRDF/CG, and SRDF/A
♦ Solutions Enabler SYMCLI TimeFinder/Mirror, TimeFinder/CG, TimeFinder/Snap,
TimeFinder/Clone
♦ Solutions Enabler SYMCLI Storage Resource Management (SRM)
These components will be discussed later in this chapter.
SYMCLI resides on a host system to monitor and perform control operations on
Symmetrix storage units. SYMCLI commands are invoked from the host operating
system command line or via scripts. SYMCLI commands invoke low-level channel
commands to specialized devices on the Symmetrix called gatekeepers. Gatekeepers are
very small devices carved from disks in the Symmetrix that act as SCSI targets for the
SYMCLI commands.
SYMCLI is used in single command line entries or in scripts to monitor and perform
control operations on devices and data objects toward the management of the storage
complex. It also monitors device configuration and status of devices that make up the
storage environment. To reduce the number of inquiries from the host to the Symmetrix
units, configuration and status information is maintained in a host database file.
SYMCLI base commands discussed in this document are listed in Table 2-1.
Table 2-1

SYMCLI base commands
Command

Argument

symdg

Description
Performs operations on a device
group (dg)

create

Creates an empty device group

delete

Deletes a device group

rename

Renames a device group

release

Releases a device external lock
associated with all devices in a
device group

list

Displays a list of all device groups
known to this host

show

Shows detailed information about a
device group and any gatekeeper
or BCV devices associated with the
device group

symcg

Performs operations on a
composite group (cg)
create

Creates an empty composite group

add

Adds a device to a composite
group

remove

Removes a device from a
composite group

EMC Solutions Enabler base management

2-5

EMC Foundation Products

delete

Deletes a composite group

rename

Renames a composite group

release

Releases a device external lock
associated with all devices in a
composite group

hold

Hold devices in a composite group

unhold

Unhold devices in a composite
group

list

Displays a list of all composite
groups known to this host

show

Shows detailed information about a
composite group, and any
gatekeeper or BCV devices
associated with the composite
group

symld

Performs operations on a device in
a device group (dg)
add

Adds devices to a device group
and assigns the device a logical
name

list

Lists all devices in a device group
and any associated BCV devices

remove

Removes a device from a device
group

rename

Renames a device in the device
group

show

Shows detailed information about a
device in a device group

symbcv

Performs support operations on
BCV pairs
list

Lists BCV devices

associate

Associates BCV devices to a
device group – required to perform
operations on the BCV device

disassociate

Disassociates BCV devices from a
device group

associate –rdf

Associates remotely attached BCV
devices to a RDF device group

disassociate –rdf

Disassociates remotely attached
BCV devices from an RDF device
group

2.3 Change Tracker
The EMC Symmetrix Change Tracker software is also part of the base Solutions Enabler
SYMCLI management offering. Change Tracker commands are used to measure
changes to data on a Symmetrix volume or group of volumes. Change Tracker
functionality is often used as a planning tool in the analysis and design of configurations
that use the EMC SRDF and TimeFinder components to create copies of production
data.

2-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

The Change Tracker command (symchg) is used to monitor the amount of changes to a
group of hypervolumes. The command timestamps and marks specific volumes for
tracking and maintains a bitmap to record which tracks have changed on those volumes.
The bitmap can be interrogated to gain an understanding of how the data on the volume
changes over time and to assess the locality of reference of applications.

2.4 EMC Symmetrix Remote Data Facility
The Symmetrix Remote Data Facility (SRDF) component of EMC Solutions Enabler
extends the basic SYMCLI command set to enable users to manage SRDF. SRDF is a
business continuity solution that provides a host-independent, mirrored data storage
solution for duplicating production site data to one or more physically separated target
Symmetrix systems. In basic terms, SRDF is a configuration of multiple Symmetrix
units whose purpose is to maintain multiple copies of logical volume data in more than
one location.
SRDF replicates production or primary (source) site data to a secondary (target) site
transparently to users, applications, databases, and host processors. The local SRDF
device, known as the source (R1) device, is configured in a partner relationship with a
remote target (R2) device, forming an SRDF pair. While the R2 device is mirrored with
the R1 device, the R2 device is write-disabled to the remote host. After the R2 device
synchronizes with its R1 device, the R2 device can be split from the R1 device at any
time, making the R2 device fully accessible again to its host. After the split, the target
(R2) device contains valid data and is available for performing business continuity tasks
through its original device address.
SRDF requires configuration of specific source Symmetrix volumes (R1) to be mirrored
to target Symmetrix volumes (R2). If the primary site is no longer able to continue
processing when SRDF is operating in synchronous mode, data at the secondary site is
current up to the last I/O transaction. When primary systems are down, SRDF enables
fast failover to the secondary copy of the data so that critical information becomes
available in minutes. Business operations and related applications may resume full
functionality with minimal interruption.
Figure 2-1 illustrates a basic SRDF configuration. As shown in the figure, connectivity
between the two Symmetrix is provided using ESCON, Fibre Channel, or Gigabit
Ethernet. The connection between the R1 and R2 volumes is through a logical grouping
of devices called an RA group. The RA group is independent of the device and
composite groups defined and discussed in Section 2.4.3.

EMC Symmetrix Remote Data Facility

2-7

EMC Foundation Products

Figure 2-1

Basic synchronous SRDF configuration

2.4.1 SRDF benefits
SRDF offers the following features and benefits:
♦ High data availability
♦ High performance
♦ Flexible configurations
♦ Host and application software transparency
♦ Automatic recovery from a component or link failure
♦ Significantly reduced recovery time after a disaster
♦ Increased integrity of recovery procedures
♦ Reduced backup and recovery costs
♦ Reduced disaster recovery complexity, planning, testing, etc.
♦ Support Business Continuance across and between multiple databases on multiple
servers and Symmetrix systems.

2.4.2 SRDF modes of operation
SRDF currently supports the following modes of operation:
♦ Synchronous mode (SRDF/S) provides real-time mirroring of data between the
source Symmetrix system(s) and the target Symmetrix system(s). Data is written

2-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

simultaneously to the cache of both systems in real time before the application I/O is
completed, thus ensuring the highest possible data availability. Data must be
successfully stored in both the local and remote Symmetrix units before an
acknowledgment is sent to the local host. This mode is used mainly for metropolitan
area network distances less than 200 km.
♦ Asynchronous mode (SRDF/A) maintains a dependent-write consistent copy of data
at all times across any distance with no host application impact. Customers wanting
to replicate data across long distances historically have had limited options.
SRDF/A delivers high-performance, extended-distance replication and reduced
telecommunication costs while leveraging existing management capabilities with no
host performance impact.
♦ Adaptive copy mode transfers data from source devices to target devices regardless
of order or consistency, and without host performance impact. This is especially
useful when transferring large amounts of data during data center migrations,
consolidations, and in data mobility environments. Adaptive copy mode is the data
movement mechanism of the Symmetrix Automated Replication solution.

2.4.3 SRDF device and composite groups
Applications running on Symmetrix arrays normally involve a number of Symmetrix
devices. Therefore, any Symmetrix operation must ensure all related devices are
operated upon as a logical group. Defining device or composite groups achieves this.
A device group is a user-defined group of devices that SYMCLI commands can execute
upon atomically. Device groups are limited to a single Symmetrix array and RA group.
A composite group, on the other hand, can span multiple Symmetrix arrays and RA
groups. The device or composite group type may contain R1 or R2 devices and may
contain various device lists for standard, BCV, virtual, and remote BCV devices. The
symdg and symcg commands are used to create and manage device and composite
groups, respectively.

2.4.4 SRDF consistency groups
An SRDF consistency group is collection of devices defined by a composite group
enabled for consistency. Its purpose is to protect data integrity for applications that span
multiple RA groups and/or multiple Symmetrix arrays. The protected applications may
comprise multiple heterogeneous data resource managers across multiple host operating
systems.
An SRDF consistency group uses PowerPath support to provide synchronous disaster
restart with zero data loss. Disaster restart solutions that use Consistency groups provide
remote restart with short recovery time objectives. Zero data loss implies that all
completed transactions at the beginning of a disaster will be available at the target.
When the amount of data for an application becomes very large, the time and resources
required for host-based software to protect, back up, or run decision-support queries on
these databases become critical factors. The time required to quiesce or shut down the
application for offline backup is no longer acceptable. SRDF consistency groups allow
users to remotely mirror the largest data environments and automatically split off

EMC Symmetrix Remote Data Facility

2-9

EMC Foundation Products

dependent-write consistent, restartable copies of applications in seconds without
interruption to online service.
A consistency group is a composite group of SRDF devices (R1 or R2) that act in unison
to maintain the integrity of applications distributed across multiple Symmetrix units or
multiple RA groups within a single Symmetrix. If a source (R1) device in the
consistency group cannot propagate data to its corresponding target (R2) device, EMC
software suspends data propagation from all R1 devices in the consistency group, halting
all data flow to the R2 targets. This suspension, called tripping the consistency group,
ensures that a dependent-write consistent R2 copy of the database up to the point in time
that the consistency group tripped.
Tripping a consistency group can occur either automatically or manually. Scenarios in
which an automatic trip would occur include:
♦ One or more R1 devices cannot propagate changes to their corresponding R2
devices
♦ The R2 device fails
♦ The SRDF directors on the R1 side or R2 side fail
In an automatic trip, the Symmetrix unit completes the write to the R1 device, but
indicates that the write did not propagate to the R2 device. EMC software intercepts the
I/O and instructs the Symmetrix to suspend all R1 source devices in the consistency
group from propagating any further writes to the R2 side. Once the suspension is
complete, writes to all of the R1 devices in the consistency group continue normally, but
they are not propagated to the target side until normal SRDF mirroring resumes.
An explicit trip occurs when the command symrdf –cg suspend or split is invoked.
Suspending or splitting the consistency group creates an on-demand, restartable copy of
the database at the R2 target site. BCV devices that are synchronized with the R2
devices are then split after the consistency group is tripped, creating a second dependentwrite consistent copy of the data. During the explicit trip, SYMCLI issues the command
to create the dependent-write consistent copy, but may require assistance from
PowerPath if I/O is received on one or more R1 devices, or if the SYMCLI commands
issued are abnormally terminated before the explicit trip.
An EMC consistency group maintains consistency within applications spread across
multiple Symmetrix units in an SRDF configuration, by monitoring data propagation
from the source (R1) devices in a consistency group to their corresponding target (R2)
devices as depicted in Figure 2-2. Consistency groups provide data integrity protection
during a rolling disaster.
Figure 2-2 depicts a dependent-write I/O sequence where a predecessor log write
happens before a page flush from a database buffer pool. The log device and data device
are on different Symmetrix arrays with different replication paths. Figure 2-2
demonstrates how rolling disasters can be prevented using EMC consistency group
technology.

2-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

Figure 2-2

SRDF consistency group

A consistency group protection is defined containing volumes X, Y, and Z on the source
Symmetrix. This consistency group definition must contain all of the devices that need
to maintain dependent-write consistency and reside on all participating hosts involved in
issuing I/O to these devices. A mix of CKD (mainframe) and FBA (UNIX/Windows)
devices can be logically grouped together. In some cases, the entire processing
environment may be defined in a consistency group to ensure dependent-write
consistency.
The rolling disaster described previously begins, preventing the replication of changes
from volume Z to the remote site.
Since the predecessor log write to volume Z cannot be propagated to the remote
Symmetrix system, a consistency group trip occurs.
A ConGroup trip holds the write that could not be replicated along with all of the writes
to the logically grouped devices. The writes are held by PowerPath on UNIX/Windows
hosts, and IOS on the mainframe hosts long enough to issue two I/Os to all of the
Symmetrix arrays involved in the consistency group. The first I/O changes the state of
the devices to a suspend-pending state.
The second I/O performs the suspend actions on the R1/R2 relationships for the logically
grouped devices which immediately disables all replication to the remote site. This
allows other devices outside of the group to continue replicating, provided the
communication links are available. After the relationship is suspended, the completion
of the predecessor write is acknowledged back to the issuing host. Furthermore, all I/Os
that were held during the consistency group trip operation are released.

EMC Symmetrix Remote Data Facility

2-11

EMC Foundation Products

After the second I/O per Symmetrix completes, the I/O is released, allowing the
predecessor log write to complete to the host. The dependent data write is issued by the
DBMS and arrives at X but is not replicated to the R2(X).
When a complete failure occurs from this rolling disaster, the dependent-write
consistency at the remote site is preserved. If a complete disaster does not occur and the
failed links are activated again, the consistency group replication can be resumed. It is
recommended to create a copy of the dependent-write consistent image while the resume
takes place. Once the SRDF process reaches synchronization the dependent-write
consistent copy is achieved at the remote site.

2.4.5 SRDF terminology
This section describes various terms related to SRDF operations.
2.4.5.1 Suspend and resume operations

Practical uses of suspend and resume operations usually involve unplanned situations in
which an immediate suspension of I/O between the R1 and R2 devices over the SRDF
links is desired. In this way, data propagation problems can be stopped. When suspend
is used with consistency groups, immediate backups can be performed off the R2s
without affecting I/O from the local host application. I/O can then be resumed between
the R1 and R2 and return to normal operation.
2.4.5.2 Establish and split operations

The establish and split operations are normally used in planned situations in which use
of the R2 copy of the data is desired without interfering with normal write operations to
the R1 device. Splitting a point-in-time copy of data allows access to the data on the R2
device for various business continuity tasks. The ease of splitting SRDF pairs to provide
exact database copies makes it convenient to perform scheduled backup operations,
reporting operations, or new application testing from the target Symmetrix data while
normal processing continues on the source Symmetrix system.
The R2 copy can also be used to test disaster recovery plans without manually intensive
recovery drills, complex procedures, and application service interruptions. Upgrades to
new versions can be tested or changes to actual code can be made without affecting the
online production server. For example, modified server code can be run on the R2 copy
of the database until the upgraded code runs with no errors before upgrading the
production server.
In cases where an absolute real-time copy of the production data is not essential, users
may choose to split the SRDF pair periodically and use the R2 copy for queries and
report generation. The SRDF pair can be reestablished periodically to provide
incremental updating of data on the R2 device. The ability to refresh the R2 device
periodically provides the latest information for data processing and reporting.
2.4.5.3 Failover and failback operations

Practical uses of failover and failback operations usually involve the need to switch
business operations from the production site to a remote site (failover) or the opposite

2-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

(failback). Once failover occurs, normal operations continue using the remote (R2) copy
of synchronized application data. Scheduled maintenance at the production site is one
example of where failover to the R2 site might be needed.
Testing of disaster recovery plans is the primary reason to temporarily fail over to a
remote site. Traditional disaster recovery routines involve customized software and
complex procedures. Offsite media must be either electronically transmitted or
physically shipped to the recovery site. Time-consuming restores and the application of
logs usually follow. SRDF failover/failback operations significantly reduce the recovery
time by incrementally updating only the specific tracks that have changed; this
accomplishes in minutes what might take hours for a complete load from dumped
database volumes.
2.4.5.4 Update operation

The update operation allows users to resynchronize the R1s after a failover while
continuing to run application and database services on the R2s. This function helps
reduce the amount of time that a failback to the R1 side takes. The update operation is a
subset of the failover/failback functionality. Practical uses of the R1 update operation
usually involve situations in which the R1 becomes almost synchronized with the R2
data before a failback, while the R2 side is still online to its host. The -until option,
when used with update, specifies the target number of invalid tracks that are allowed to
be out of sync before resynchronization to the R1 completes.
2.4.5.5 Concurrent SRDF

Concurrent SRDF means having two target R2 devices configured as concurrent mirrors
of one source R1 device. Using a concurrent SRDF pair allows the creation of two
copies of the same data at two remote locations. When the two R2 devices are split from
their source R1 device, each target site copy of the application can be accessed
independently.
2.4.5.6 R1/R2 swap

Swapping R1/R2 devices of an SRDF pair causes the source R1 device to become a
target R2 device and vice versa. Swapping SRDF devices allows the R2 site to take over
operations while retaining a remote mirror on the original source site. Swapping is
especially useful after failing over an application from the R1 site to the R2 site. SRDF
swapping is available with Enginuity version 5567 or higher.
2.4.5.7 Data Mobility

Data Mobility is an SRDF configuration that restricts SRDF devices to operating only in
adaptive copy mode. This is a lower-cost licensing option that is typically used for data
migrations. It allows data to be transferred asynchronously from source to target, and is
not designed as a solution for DR requirements unless used in combination with
TimeFinder.

EMC Symmetrix Remote Data Facility

2-13

EMC Foundation Products

2.4.5.8 Dynamic SRDF

Dynamic SRDF allows the creation of SRDF pairs from non-SRDF devices while the
Symmetrix unit is in operation. Historically, source and target SRDF device pairing has
been static and changes required assistance from EMC personnel. However, beginning
with the SRDF component of EMC Solutions Enabler version 5.0 running on Symmetrix
units using Enginuity™ version 5568, users can use SRDF-capable, non-SRDF devices in
creating and synchronizing SRDF pairs. This feature provides greater flexibility in
deciding where to copy protected data.
Beginning with Solutions Enabler version 5.2 running on Symmetrix units using
Enginuity version 5669, dynamic RA groups can be created in a SRDF switched fabric
environment. An RA group represents a logical connection between two Symmetrix
units. Historically, RA groups were limited to those static RA groups defined at
configuration time. However, RA groups can now be created, modified, and deleted
while the Symmetrix unit is in operation. This feature provides greater flexibility in
forming SRDF-pair-associated links.

2.4.6 SRDF control operations
This section describes typical control operations that can be performed by the Solutions
Enabler symrdf command.
Solutions Enabler SYMCLI SRDF commands perform the following basic control
operations on SRDF devices:
♦ Establish synchronizes an SRDF pair by initiating a data copy from the source (R1)
side to the target (R2) side. This operation can be a full or incremental establish.
Changes on the R2 volumes are discarded by this process.
♦ Restore resynchronizes a data copy from the target (R2) side to the source (R1) side.
This operation can be a full or incremental restore. Changes on the R1 volumes are
discarded by this process.
♦ Split stops mirroring for the SRDF pair(s) in a device group and write-enables the
R2 devices.
♦ Swap exchanges the source (R1) and target (R2) designations on the source and
target volumes.
♦ Failover switches data processing from the source (R1) side to the target (R2) side.
The source side volumes (R1), if still available, are write-disabled.
♦ Failback switches data processing from the target (R2) side to the source (R1) side.
The target side volumes (R2), if still available, are write-disabled.
2.4.6.1 Establishing an SRDF pair

Establishing an SRDF pair initiates remote mirroring—the copying of data from the
source (R1) device to the target (R2) device. SRDF pairs come into existence in two
different ways:

2-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

♦ At configuration time through the pairing of SRDF devices. This is a static pairing
configuration discussed earlier.
♦ Anytime during a dynamic pairing configuration in which SRDF pairs are created on
demand.
A full establish (symrdf establish –full) is typically performed after an SRDF pair is
initially configured and connected via the SRDF links. After the first full establish,
users can perform an incremental establish, where the R1 device copies to the R2 device
only the new data that was updated while the relationship was split or suspended.
To initiate an establish operation on all SRDF pairs in a device or composite group, all
pairs must be in the split or suspended state. The symrdf query command is used to
check the state of SRDF pairs in a device or composite group.
When the establish operation is initiated, the system write-disables the R2 device to its
host and merges the track tables. The merge creates a bitmap of the tracks that need to
be copied to the target volumes discarding the changes on the target volumes. When the
establish operation is complete, the SRDF pair is in the synchronized state. The R1
device and R2 device contain identical data, and continue to do so until interrupted by
administrative command or unplanned disruption. Figure 2-3 depicts SRDF establish
and restore operations:

Figure 2-3

SRDF establish and restore control operations

The establish operation may be initiated by any host connected to either Symmetrix
array, provided that an appropriate device group has been built on that host. The
following command initiates an incremental establish operation for all SRDF pairs in the
device group named device_group:
symrdf –g device_group establish –noprompt

EMC Symmetrix Remote Data Facility

2-15

EMC Foundation Products

2.4.6.2 Splitting an SRDF pair

When read/write access to a target (R2) device is necessary, the SRDF pair can be split.
When the split completes, the target host can access the R2 device for write operations.
The R2 device contains valid data and is available for business continuity tasks or
restoring data to the R1 device if there is a loss of data on that device.
While an SRDF pair is in the split state, local I/O to the R1 device can still occur. These
updates are not propagated to the R2 device immediately. Changes on each Symmetrix
system are tracked through bitmaps and are reconciled when normal SRDF mirroring
operations are resumed. To initiate a split, an SRDF pair must already be in one of the
following states:
♦ Synchronized
♦ Suspended
♦ R1 updated
♦ SyncInProg (if the –symforce option specified for the split)
The split operation may be initiated from either host. The following command initiates a
split operation on all SRDF pairs in the device group named device_group:
symrdf –g device_group split –noprompt

The symrdf split command provides exactly the same functionality as the symrdf
suspend and symrdf rw_enable R2 commands together. Furthermore, the split and
suspend operations have exactly the same consistency characteristics as SRDF
consistency groups. Therefore, when SRDF pairs are in a single device group, users can
split the SRDF pairs in the device group as shown previously and have restartable copies
on the R2 devices. If the application data spans multiple Symmetrix units or multiple
RA groups, include SRDF pairs in a consistency group to achieve the same results.
2.4.6.3 Restoring an SRDF pair

When the target (R2) data must be copied back to the source (R1) device, the SRDF
restore command is used (see Figure 2-3). After an SRDF pair is split, the R2 device
contains valid data and is available for business continuance tasks (such as running a
new application) or restoring data to the R1 device. Moreover, if the results of running a
new application on the R2 device need to be preserved, moving the changed data and
new application to the R1 device is another option.
Users may perform a full or incremental restore. A full restore operation copies the
entire contents of the R2 device to the R1 device. An incremental restore operation is
much quicker because it copies only new data that was updated on the R2 device while
the SRDF pair was split. Any tracks on the R1 device that changed while the SRDF pair
was split are replaced with the corresponding tracks on the R2 device.
To initiate a restore, an SRDF pair must already be in the split state. The restore
operation can be initiated from either host. The following command initiates an
incremental restore operation on all SRDF pairs in the device group named
device_group (add the –full option for a full restore):
2-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

symrdf –g device_group restore –noprompt

The restore operation is complete when the R1 and R2 devices contain identical data.
The SRDF pair is then in a synchronized state and may be reestablished by initiating the
symrdf establish command.

2.4.7 Failover and failback operations
Having a synchronized SRDF pair allows users to switch data processing operations
from the source site to the target site if operations at the source site are disrupted or if
downtime must be scheduled for maintenance. This switchover from source to target is
called failover. When operations at the source site are back to normal, a failback
operation is used to reestablish I/O communications links between source and target,
resynchronize the data between the sites, and resume normal operations on the R1
devices as shown in Figure 2-4.
Figure 2-4 illustrates the failover and failback operations. The failover and failback
operations relocate the processing from the source site to the target site or vice versa.
This may or may not imply movement of data.

Figure 2-4

SRDF failover and failback control operations

2.4.7.1 Failover

Scheduled maintenance or storage system problems can disrupt access to production
data at the source site. In this case, a failover operation can be initiated from either host
to make the R2 device read/write-enabled to its host. Before issuing the failover, all
applications services on the R1 volumes must be stopped. This is because the failover
operation will make the R1 volumes read-only. The following command initiates a
failover on all SRDF pairs in the device group device_group:

EMC Symmetrix Remote Data Facility

2-17

EMC Foundation Products

symrdf –g device_group failover –noprompt

To initiate a failover, the SRDF pair must already be in one of the following states:
♦ Synchronized
♦ Suspended
♦ R1 updated
♦ Partitioned (when invoking this operation at the target site)
The failover operation:
♦ Suspends data traffic on the SRDF links
♦ Write-disables the R1 devices
♦ Write-enables the R2 volumes.
2.4.7.2 Failback

To resume normal operations on the R1 side, a failback (R1 device takeover) operation
is initiated. This means read/write operations on the R2 device must be stopped, and
read/write operations on the R1 device must be started. When the failback command is
initiated, the R2 becomes read-only to its host, while the R1 becomes read/write-enabled
to its host. The following command performs a failback operation on all SRDF pairs in
the device group device_group:
symrdf –g device_group failback -noprompt

The SRDF pair must already be in one of the following states for the failback operation
to succeed:
♦ Failed over
♦ Suspended and write-disabled at the source
♦ Suspended and not ready at the source
♦ R1 Updated
♦ R1 UpdInProg
The failback operation:
♦ Write-enables the R1 devices.
♦ Performs a track table merge to discard changes on the R1s.
♦ Transfers the changes on the R2s.
♦ Resumes traffic on the SRDF links.
♦ Write-disables the R2 volumes.

2-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

2.4.8 SRDF/A (Asynchronous) operations
SRDF Asynchronous (SRDF/A) operations are supported with EMC Solutions Enabler
SYMCLI 5.3 and higher. Symmetrix DMX arrays running Enginuity 5670 code and
higher support SRDF/A mode for SRDF devices. Asynchronous mode provides a
dependent-write consistent, point-in-time image on the target (R2) devices, which are
only slightly behind the source (R1) device. SRDF/A session data is transferred to the
remote Symmetrix array in predefined timed cycles or delta sets. This functionality
requires an SRDF/A license.
The default configuration for SRDF/A is for the target volumes to be at most two cycles behind
the data state of the source volumes. In the default configuration for SRDF/A, this would equate
to 60 seconds.

SRDF/A provides a long-distance replication solution with minimal impact on
performance that preserves data consistency within application data in the specified
business process. This level of protection is intended for DR environments that always
need a DBMS restartable copy of data at the R2 site. Dependent-write consistency is
guaranteed on a delta set boundary. In the event of a disaster at the R1 site or if SRDF
links are lost during data transfer, a partial delta set of data will be discarded. This
preserves consistency on the R2 with a maximum data loss of two SRDF/A cycles.
2.4.8.1 Enabling SRDF/A

To set remotely mirrored pairs in the prod group to the asynchronous mode (SRDF/A),
enter:
symrdf –g device_group set mode async –noprompt
2.4.8.2 SRDF/A benefits

SRDF/A mode provides the following features and benefits:
♦ Lowers operational costs for long-distance data replication with application
consistency.
♦ Promotes efficient link utilization, resulting in lower link bandwidth.
♦ Maintains a dependent-write consistent, point-in-time image on the R2 devices at all
times.
♦ Supports all current SRDF topologies, including point-to-point and switched fabric.
♦ Operates at any given distance without adding response time to the R1 host.
♦ Includes all hosts and data emulation types supported by the Symmetrix array (such
as FBA, CKD, and AS/400).
♦ Minimizes the impact imposed on the back-end directors.
♦ Provides an application response time equivalent to writing to local non-SRDF
devices.
♦ Allows restore, failover, and failback capability between the R1 and R2 sites.

EMC Symmetrix Remote Data Facility

2-19

EMC Foundation Products

2.4.8.3 SRDF/A capable devices

A listing of SRDF/A capable devices is displayed with the following command:
symrdf list –rdfa
2.4.8.4 SRDF/A session status

When asynchronous mode is set for a group of devices, the SRDF/A capable devices in
the group are considered part of the SRDF/A session. To check SRDF/A session status,
use either of the following commands:
symdg show device_group
symcg show composite_group

The session status is displayed as active or inactive:
♦ Inactive — The SRDF/A devices are either ready or not ready on the link
♦ Active — The SRDF/A mode is activated and the SRDF/A session data is currently
being transmitted in operational cycles to the R2.

2.4.9 EMC SRDF/Cluster Enabler solutions
EMC SRDF/Cluster Enabler (SRDF/CE) for MSCS is an integrated solution that
combines SRDF and clustering protection over distance. EMC SRDF/CE provides
disaster-tolerant capabilities that enable a cluster to span geographically separated
Symmetrix systems. It operates as a software extension (MMC snap-in) to the Microsoft
Cluster Service (MSCS).
SRDF/CE achieves this capability by exploiting SRDF disaster restart capabilities.
SRDF allows the MSCS cluster to have two identical sets of application data in two
different locations. When cluster services are failed over or failed back, SRDF/CE is
invoked automatically to perform the SRDF functions necessary to enable the requested
operation.
Figure 2-5 illustrates the hardware configuration of two, four-node, geographically
distributed EMC SRDF/CE clusters using bidirectional SRDF.

2-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

Figure 2-5

Geographically distributed four-node EMC SRDF/CE clusters

2.5 EMC TimeFinder
The SYMCLI TimeFinder component extends the basic SYMCLI command set to
include TimeFinder or business continuity commands that allow control operations on
device pairs within a local replication environment. This section specifically describes
the functionality of:
♦ TimeFinder/Mirror — General monitor and control operations for business
continuance volumes (BCV).
♦ TimeFinder/CG — Consistency groups
♦ TimeFinder/Clone — Clone copy sessions
♦ TimeFinder/Snap — Snap copy sessions
Commands such as symmir and symbcv perform a wide spectrum of monitor and control
operations on standard/BCV device pairs within a TimeFinder/Mirror environment. The
TimeFinder/Clone command, symclone, creates a point-in-time copy of a source device
on nonstandard device pairs (such as standard/standard, BCV/BCV). The
TimeFinder/Snap command, symsnap, creates virtual device copy sessions between a
source device and multiple virtual target devices. These virtual devices only store
pointers to changed data blocks from the source device, rather than a full copy of the
data. Each product requires a specific license for monitoring and control operations.

EMC TimeFinder

2-21

EMC Foundation Products

Configuring and controlling remote BCV pairs requires EMC SRDF business continuity
software discussed in section 2.4. The combination of TimeFinder with SRDF provides
for multiple local and remote copies of production data.
Figure 2-6 illustrates Application Usage for TimeFinder/Mirror configuration in a
Symmetrix system.

Figure 2-6

EMC Symmetrix configured with standard volumes and BCVs

2.5.1 TimeFinder/Mirror Establish operations
A BCV device can be fully or incrementally established. After configuration and
initialization of a Symmetrix unit, BCV devices contain no data. BCV devices, like
standard devices, can have unique host addresses and can be online and ready to the
host(s) to which they are connected. A full establish must be used the first time the
standard devices are paired with the BCV devices. An incremental establish of a BCV
device can be performed to resynchronize any data that has changed on the standard
since the last establish.
When BCVs are established, they are inaccessible to any host.

Symmetrix arrays allow up to four mirrors for each logical volume. The mirror
positions are commonly designated M1, M2, M3, and M4. An unprotected BCV can be
the second, third, or fourth mirror position of the standard device. A host, however,
logically views the Symmetrix M1/M2 mirrored devices as a single device.
To assign a BCV as a mirror of a standard Symmetrix device, the symmir
establish command is used. One method of establishing a BCV pair is to allow the
standard/BCV device-pairing algorithm to arbitrarily create BCV pairs from multiple
devices within a device group:

2-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

symmir -g device_group establish –full -noprompt

With this method, TimeFinder/Mirror first checks for any attach assignments (specifying
a preferred BCV match from among multiple BCVs in a device group).
TimeFinder/Mirror then checks if there are any pairing relationships among the devices.
If either of these previous conditions exists, TimeFinder/Mirror uses these assignments.

2.5.2 TimeFinder split operations
Splitting a BCV pair is a TimeFinder/Mirror action that detaches the BCV from its
standard device and makes the BCV ready for host access. When splitting a BCV, the
system must perform housekeeping tasks that may require a few milliseconds on a busy
Symmetrix array. These tasks involve a series of steps that result in separation of the
BCV from its paired standard:
♦ I/O is suspended briefly to the standard device.
♦ Write pending tracks for the standard device that have not yet been written out to the
BCV are duplicated in cache to be written to the BCV.
♦ The BCV is split from the standard device.
♦ The BCV device status is changed to ready.
2.5.2.1 Regular split

A regular split is the type of split that has existed for TimeFinder/Mirror since its
inception. With a regular split (before Enginuity version 5568), I/O activity from the
production hosts to a standard volume was not accepted until it was split from its BCV
pair. Therefore, applications attempting to access the standard or the BCV would
experience a short wait during a regular split. Once the split was complete, no further
overhead was incurred.
Beginning with Enginuity version 5568, any split operation is an instant split. A
regular split is still valid for earlier versions and for current applications that perform
regular split operations. However, current applications that perform regular splits with
Enginuity version 5568 actually perform an instant split.
By specifying the –instant flag on the command line, an instant split with Enginuity
5x66 and 5x67 can be performed. Since 5568, this option is no longer required because
instant split mode has become the default behavior.
2.5.2.2 Instant split

An instant split shortens the wait period during a split by dividing the process into a
foreground split and a background split. During an instant split, the system executes the
foreground split almost instantaneously and returns a successful status to the host. This
instantaneous execution allows minimal I/O disruptions to the production volumes.
Furthermore, the BCVs are accessible to the hosts as soon as the foreground process is
complete. The background split continues to split the BCV pair until it is complete.
When the -instant option is included or defaulted, SYMCLI returns immediately after

EMC TimeFinder

2-23

EMC Foundation Products

the foreground split, allowing other operations while the BCV pair is splitting in the
background.
The following operation performs an instant split on all BCV pairs in device_group, and
allows SYMCLI to return to the server process while the background split is in progress.
symmir -g device_group split –instant –noprompt

The following symmir query command example checks the progress of a split on
composite_group. The –bg option is provided to query the status of the background
split:
symmir –cg composite_group query –bg

2.5.3 TimeFinder restore operations
A BCV device can be used to fully or incrementally restore data on the standard volume.
Like the full establish operation, a full restore operation copies the entire contents of the
BCV devices to the standard devices. Optionally, users may specify devices in a device
group, composite group, or device file as in the following examples:
symmir -g device_group -full restore –noprompt
symmir -cg composite_group -full restore –noprompt
symmir -f[ile] FileName -full –sid <###> restore -noprompt

The incremental restore process accomplishes the same thing as the full restore process
with a major time-saving exception: the BCV copies to the standard device only new
data that was updated on the BCV device while the BCV pair was split. The data on the
corresponding tracks of the BCV device also overwrites any changed tracks on the
standard device. This maximizes the efficiency of the resynchronization process. This
process is useful, for example, if, after testing or validating an updated version of a
database or a new application on the BCV device is completed, a user wants to migrate
and utilize a copy of the tested data or application on the production standard device.
An incremental restore of a BCV volume to a standard volume is only possible when the two
volumes have an existing TimeFinder relationship

2.5.4 TimeFinder consistent split
TimeFinder consistent split allows users to split off a dependent-write consistent,
restartable image of an application without interrupting online services. Consistent split
helps to avoid inconsistencies and restart problems that can occur when splitting an
application-related BCV without first quiescing or halting the application. Consistent
split is implemented using the Enginuity Consistency Assist (ECA) feature. This
functionality requires a TimeFinder/CG license.
2.5.4.1 Enginuity consistency assist

The Enginuity Consistency Assist feature of the Symmetrix operating environment can
be used to perform consistent split operations across multiple heterogeneous

2-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

environments. This functionality requires a TimeFinder/CG license and uses the –
consistent option of the symmir command.
Using ECA to consistently split BCV devices from the standards, a control host with no
database or a database host with a dedicated channel to gatekeeper devices must be
available. The dedicated channel cannot be used for servicing other devices or to freeze
I/O. For example, to split a device group, execute:
symmir –g device_group split –consistent -noprompt

Figure 2-7 illustrates an Enginuity Consistency Assist split across three database hosts
that access devices on a Symmetrix array:

Figure 2-7

ECA consistent split across multiple database associated hosts

Symmetrix device or composite groups must be created on the controlling host for the
target application to be consistently split. Device groups can be created to include all of
the required devices for maintaining business continuity. For example, if a device group
is defined that includes all of the devices being accessed by Hosts A, B, and C (see
Figure 2-7), then all of the BCV pairs related to those hosts can be consistently split with
a single command.
However, if a device group is defined that includes only the devices accessed by Host A,
then the BCV pairs related to Host A can be split without affecting the other hosts. The
solid vertical line in the in Figure 2-7 represents the ECA holding of I/Os during an
instant split process, creating a dependent-write consistent image in the BCVs.

EMC TimeFinder

2-25

EMC Foundation Products

Figure 2-8 illustrates the use of local consistent split with a database management
system (DBMS).

Figure 2-8

ECA consistent split on a local Symmetrix

When a split command is issued with ECA from the production host, a consistent
database image is created using the following sequence of events shown in Figure 2-8:
1. The Symmetrix API (SYMAPI) identifies the standard devices that hold the
database.
2. SYMAPI validates that all identified BCV pairs can be split.
3. SYMAPI sends a suspend write message to ECA.
4. ECA suspends writes to the standard devices that hold the database. The DBMS
cannot write to the devices and subsequently waits for these devices to become
available before resuming any further write activity. Read activity to the device is
not affected unless attempting to read from a device which has a write queued
against it.
5. SYMAPI sends an instant split request to all BCV pairs in the specified device
group and waits for the Symmetrix to acknowledge that the foreground split has
occurred. SYMAPI then sends a resume I/O message to ECA.
6. The application resumes writing to the production devices.
The BCV devices now contain a restartable copy of the production data that is consistent
up until the time of the instant split. The production application is unaware that the split
or suspend/resume operation occurred. When the application on the secondary host is
started using the BCVs, there is no record of a successful shutdown. Therefore, the

2-26

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

secondary application instance views the BCV copy as a crashed instance and proceeds
to perform the normal crash recovery sequence to restart.
When performing a consistent split, it is a good practice to issue host-based commands
that will commit any data that has not been written to disk before the split. For example
on UNIX systems, the sync command can be run. From a database perspective, a
checkpoint or equivalent should be executed.

2.5.5 TimeFinder reverse split
BCVs can be mirrored to guard against data loss through physical drive failures. A
reverse split is applicable for a BCV that is configured to have two local mirrors. It is
generally used to recover from an unsuccessful restore operation. When data is restored
from the BCV to the standard device, any writes that occur while the standard is being
restored alter the original copy of data on the BCV’s primary mirror. If the original
copy of BCV data is needed again at a later time, it can be restored to the BCV’s
primary mirror from the BCV’s secondary mirror using a reverse split. For example,
whenever logical corruption is reintroduced to a database during a recovery process
(following a BCV restore), both the standard device and the primary BCV mirror are left
with corrupted data. In this case, a reverse split can restore the original BCV data from a
BCV’s secondary mirror to its primary mirror.
This is particularly useful when performing a restore and immediately restarting
processing on the standard devices when the process may have to be restarted many
times.
Reverse split is not available when protected restore is used to return the data from the BCVs to
the standards.

2.5.6 TimeFinder/Clone operations
Symmetrix TimeFinder/Clone operations using SYMCLI have been available since
Enginuity version 5568. TimeFinder/Clone can create up to 16 copies from a source
device onto target devices. Unlike TimeFinder/Mirror, TimeFinder/Clone does not
require the traditional standard to BCV device pairing. Instead, TimeFinder/Clone
allows any combination of source and target devices. For example, a BCV can be used
as the source device, while another BCV can be used as the target device. Any
combination of source and target devices can be used. Additionally, TimeFinder/Clone
does not use the traditional mirror positions the way that TimeFinder/Mirror does.
Because of this, TimeFinder/Clone is a useful option when more than three copies of a
source device are desired.
Normally, one of the three copies is used to protect the data against hardware failure.
The source and target devices must be the same emulation type (FBA or CKD). The
target device must be equal or greater in size than the source. Clone copies of striped or
concatenated metavolumes can also be created providing the source, and target
metavolumes must be identical in configuration. Once activated, the target device can
be instantly accessed by a target’s host, even before the data is fully copied to the target
device.

EMC TimeFinder

2-27

EMC Foundation Products

TimeFinder/Clone copies are appropriate in situations where multiple copies of
production data is needed for testing, backups, or report generation. Clone copies can
also be used to reduce disk contention and improve data access speed by assigning users
to copies of data rather than accessing the one production copy. A single source device
may maintain as many as 16 relationships that can be a combination of BCV’s, clones
and snaps. When using the -copy option of TimeFinder/Clones, one can copy up to four
full data copies simultaneously, without disruption to database production activity.
2.5.6.1 Clone copy sessions

TimeFinder/Clone functionality is controlled via copy sessions, which pair the source
and target devices. Sessions are maintained on the Symmetrix array and can be queried
to verify the current state of the device pairs. A copy session must first be created to
define and set up the TimeFinder/Clone devices. The session is then activated, enabling
the target device to be accessed by its host. When the information is no longer needed,
the session can be terminated. TimeFinder/Clone operations are controlled from the host
by using the symclone command to create, activate, and terminate the copy sessions.
Figure 2-9 illustrates a copy session where the controlling host creates a
TimeFinder/Clone copy of standard device DEV001 on target device DEV005, using the
symclone command.

Figure 2-9

Creating a copy session using the symclone command

The symclone command is used to enable cloning operations. The cloning operation
happens in two phases: creation and activation. The creation phase builds bitmaps of the
source and target that are later used during the activation or copy phase. The creation of
a symclone pairing does not start copying of the source volume to the target volume. To
create clone sessions on all the standards and BCVs in the device group device_group,
use the following command:

2-28

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

symclone -g device_group create -noprompt

The activation of a clone enables the copying of the data. The data may start copying
immediately if the –copy keyword is used. If the –copy keyword is not used, tracks are
only copied when they are accessed from the target volume or when they are changed on
the source volume.
Activation of the clone session established in the previous create command can be
accomplished using the following command:
symclone –g device_group activate -noprompt

2.5.7 TimeFinder/Snap operations
Symmetrix DMX arrays running 5669 or later version of the Enginuity operating
environment provide another technique to create copies of application data. The
functionality, called TimeFinder/Snap, allows users to make pointer-based, space-saving
copies of data simultaneously on multiple target devices from a single source device.
The data is available for access instantly. TimeFinder/Snap allows data to be copied
from a single source device to as many as 15 target devices (a sixteenth Snap session is
reserved for restoring data from one of the targets). A source device can be either a
Symmetrix standard device or a BCV device controlled by TimeFinder/Mirror. The
target device is a Symmetrix virtual device (VDEV) that consumes negligible physical
storage through the use of pointers to track changed data.
A VDEV is a host-addressable Symmetrix device with special attributes created when
the Symmetrix DMX system is configured. However, unlike a BCV which contains a
full volume of data, a VDEV is a logical-image device that offers a space-saving way to
create instant, point-in-time copies of volumes. Any updates to a source device after its
activation with a virtual device, causes the pre-update image of the changed tracks to be
copied to a save device. The virtual device’s indirect pointer is then updated to point to
the original track data on the save device, preserving a point-in-time image of the
volume. TimeFinder/Snap uses this copy-on-first-write technique to conserve disk
space, since only changes to tracks on the source cause any incremental storage to be
consumed.
The symsnap create and symsnap activate are used to create source/target Snap pair.
Table 2-2 summarizes some of the differences between devices used in
TimeFinder/Snap operations.
Table 2-2

TimeFinder device type summary
Device

Description

Virtual device

A logical-image device that saves disk space through the use of pointers
to track data that is immediately accessible after activation. Snapping
data to a virtual device uses a copy-on-first-write technique.

Save device

A device that is not host-accessible but accessed only through the virtual
devices that point to it. Save devices provide a pool of physical space to
store snap copy data to which virtual devices point.

BCV

A full volume mirror that has valid data after fully synchronizing with its
source device. It is accessible only when split from the source device

EMC TimeFinder

2-29

EMC Foundation Products

that it is mirroring.

2.5.7.1 Snap copy sessions

TimeFinder/Snap functionality is managed via copy sessions, which pair the source and
target devices. Sessions are maintained on the Symmetrix array and can be queried to
verify the current state of the devices. A copy session must first be created—a process
which defines the Snap devices in the operation. On subsequent activation, the target
virtual devices become accessible to its host. Unless the data is changed by the host
accessing the virtual device, the virtual device always presents a frozen point-in-time
copy of the source device at the point of activation. When the information is no longer
needed, the session can be terminated.
TimeFinder/Snap operations are controlled from the host by using the symsnap
command to create, activate, terminate, and restore the TimeFinder/Snap copy sessions.
The TimeFinder/Snap operations described in this section explain how to manage the
devices participating in a copy session through SYMCLI.
Figure 2-10 illustrates a virtual copy session where the controlling host creates a copy of
standard device DEV001 on target device VDEV005.

Figure 2-10 TimeFinder/Snap copy of a standard device to a VDEV

The symsnap command is used is used to enable TimeFinder/Snap operations. The Snap
operation happens in two phases: creation and activation. The creation phase builds
bitmaps of the source and target that are later used to manage the changes on the source
and target. The creation of a snap pairing does not copy the data from the source volume
to the target volume. To create snap sessions on all the standards and BCVs in the
device group device_group, use the following command:
symsnap -g device_group create -noprompt

2-30

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

The activation of a snap enables the protection of the source data tracks. When
protected tracks are changed on the source volume, they are first copied into the save
pool and the VDEV pointers are updated to point to the changed tracks in the save pool.
When tracks are changed on the VDEV, the data is written directly to the save pool and
the VDEV pointers are updated in the same way.
Activation of the snap session created in the previous create command can be
accomplished using the following command:
symsnap –g device_group activate -noprompt

2.6 EMC Storage Resource Management
The Storage Resource Management (SRM) component of EMC Solutions Enabler
extends the basic SYMCLI command set to include SRM commands that allow users to
discover and examine attributes of various objects on a host or in the EMC storage
enterprise.
SYMCLI commands support SRM in the following areas:
♦ Data objects and files
♦ Relational databases
♦ File systems
♦ Logical volumes and volume groups
♦ Performance statistics
SRM allows users to examine the mapping of storage devices and the characteristics of
data files and objects. These commands allow the examination of relationships between
extents and data files or data objects, and how they are mapped on storage devices.
Frequently, SRM commands are used with TimeFinder and SRDF to create point-intime copies for backup and restart.
Figure 2-11 outlines the process of how SRM commands are used with TimeFinder in a
database environment.

EMC Storage Resource Management

2-31

EMC Foundation Products

Figure 2-11 SRM commands

EMC Solutions Enabler with a valid license for TimeFinder and SRM is installed on the
host. In addition, the host must also have PowerPath or use ECA, and must be utilized
with a supported DBMS system. As discussed in section 2.5.2, when splitting a BCV,
the system must perform housekeeping tasks that may require a few seconds on a busy
Symmetrix array. These tasks involve a series of steps (shown in Figure 2-11) that result
in the separation of the BCV from its paired standard:
1. Using the SRM base mapping commands, first query the Symmetrix unit to display
the logical-to-physical mapping information about any physical device, logical
volume, file, directory, and/or file system.
2. Using the database mapping command, query the Symmetrix to display physical and
logical database information.
3. Next, use the database mapping command to translate:
•

The devices of a specified database into a device group or a consistency group,
or

•

The devices of a specified tablespace into a device group or a consistency group.

4. The BCV is split from the standard device.

2-32

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

Table 2-3 lists the SYMCLI commands that can be used to examine the mapping of data
objects.

EMC Storage Resource Management

2-33

EMC Foundation Products

Table 2-3

Data object SRM commands
Command

Argument

Action

symrslv

pd

Displays logical to physical mapping information
about any physical device.

lv

Displays logical to physical mapping information
about a logical volume.

file

Displays logical to physical mapping information
about a file.

dir

Displays logical to physical mapping information
about a directory.

fs

Displays logical to physical mapping information
about a file system.

SRM commands allow users to examine the host database mapping and the
characteristics of a database. The commands provide listings and attributes that describe
various databases, their structures, files, tablespaces, and user schemas. Typically, the
database commands work with Oracle, Informix, SQL Server, Sybase, Microsoft
Exchange, SharePoint Portal Server, and DB2 LUW database applications.
Table 2-4 lists the SYMCLI commands that can be used to examine the mapping of
database objects.
Table 2-4

Data object mapping commands
Command

Argument

Action

symrdb

list

Lists various physical and logical
database objects:
Current relational database instances
available
Tablespaces, tables, files, or schemas
of a database
Files, segments, or tables of a
database tablespace or schema

show

Shows information about a database
object:
Tablespace, tables, file, or schema of
a database
File, segment, or a table of a
specified tablespace or schema

2-34

rdb2dg

Translates the devices of a specified
database into a device group.

rdb2cg

Translates the devices of a specified
tablespace into a composite group or a
consistency group.

tbs2cg

Translates the devices of a specified
tablespace into a composite group.
Only data database files are
translated.

tbs2dg

Translates the devices of a specified

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

tablespace into a device group. Only
data database files are translated.

The SYMCLI file system SRM command allows users to investigate the file systems
that are in use on the operating system. The command provides listings and attributes
that describe file systems, directories, and files, and their mapping to physical devices
and extents.
Table 2-5 lists the SYMCLI command that can be used to examine the file system
mapping.
Table 2-5

File system SRM command
Command

Argument

Action

symhostfs

list

Displays a list of file systems,
files, or directories

show

Displays more detail information about
a file system or file system object.

SYMCLI logical volume SRM commands allow users to map logical volumes to display
a detailed view of the underlying storage devices. Logical volume architecture defined
by a Logical Volume Manager (LVM) is a means for advanced applications to improve
performance by the strategic placement of data.
Table 2-6 lists the SYMCLI commands that can be used to examine the logical volume
mapping.
Table 2-6

File system SRM commands
Command

Argument

Action

symvg

deport

Deports a specified volume group so it
can be imported later.

import

Imports a specified volume group.

list

Displays a list of volume groups
defined on the host system by the
logical volume manager.

rescan

Rescans all the volume groups.

show

Displays more detail information about
a volume group.

vg2cg

Translates volume groups to composite
groups.

vg2dg

Translates volume groups to device
groups.

List

Displays a list of logical volumes on
a specified volume group.

show

Displays detail information (including
extent data) about a logical volume.

symlv

EMC Storage Resource Management

2-35

EMC Foundation Products

SRM performance statistics commands allow users to retrieve statistics about a host’s
CPU, disk, and memory.
Table 2-7 lists the statistics commands.
Table 2-7

SRM statistics command
Command

Argument

Action

symhost

Show

Displays host configuration
information.

Stats

Displays performance statistics.

2.7 EMC ControlCenter
EMC ControlCenter is an integrated family of products that enables users to discover,
monitor, automate, provision, and report on networks, host resources, and storage
resources across the entire information environment. Figure 2-12 presents the
ControlCenter product family and the broad scope of manageability that ControlCenter
provides in managing the storage enterprise.

Figure 2-12 ControlCenter family overview

The ControlCenter suite of products provides end-to-end management of storage
networks, storage devices, and other storage resources. From the Web Console,
ControlCenter can manage, monitor, configure and report on the following components:
♦ Storage components for Symmetrix, CLARiiON, Celerra®, HDS, HP StorageWorks,
IBM ESS, and NetApp filers

2-36

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

♦ Host components, such as logical volume managers, file systems, databases, backup
applications, and host processes
♦ Connectivity components, such as Fibre Channel switches and hubs
♦ Databases, such as Oracle, Sybase, and Microsoft SQL Server
Every physical and logical element that ControlCenter manages is known as a managed
object. From a console anywhere on the network, ControlCenter shows an integrated
and consolidated view of the storage environment, enabling the storage administrator to
monitor, report, manage, and configure each managed object.
ControlCenter is designed for use in a heterogeneous environment where information is
distributed in multivendor storage networks with disparate computing, connectivity, and
storage arrays. The ability to manage host applications and their associated storage
needs across heterogeneous environments from a single interface simplifies storage
management and makes it possible to implement cross-platform storage-wide strategies.
At the heart of ControlCenter is a common foundation and infrastructure, which
provides scalability, usability, and information sharing across all ControlCenter
applications. This design enables applications to span storage array, storage network,
and host management functions. The core components of ControlCenter are known as
the Open Integration Components. These provide the infrastructure (ECC Server,
Repository, and Store), common services, and a common user interface (both a webbased console for monitoring and a java-based application console for full management
functionality). The common services provide the following features:
♦ Discover storage arrays, connectivity devices, and hosts.
♦ Map the storage topology.
♦ Map the relationship of storage structures from databases and file systems to their
logical and physical location within the storage array.
♦ Display the properties of any object.
♦ Monitor the real-time or historical performance of objects.
♦ Monitor the status of objects and issue alerts.
The following ControlCenter applications are discussed briefly below:
♦ SAN Manager — provides integrated network discovery, topology, and alert
capabilities. It actively controls SAN management functions such as zoning and
LUN masking.
♦ Automated Resource Manager (ARM) — simplifies and automates storage resource
management (SRM) across the enterprise. It provides policy-based storage
provisioning for file systems and volume groups from a host perspective. This
includes determining the location of free storage pool and automating the allocation
of that storage to the host. ARM manages backup and restore operations from
within ControlCenter, and increases the availability of storage environments.

EMC ControlCenter

2-37

EMC Foundation Products

♦ StorageScope™ — provides a variety of SRM reports to help users assess their
current storage environment and determine future storage requirements.
StorageScope users can create custom report layouts, export reports into CSV or
XML format, and integrate StorageScope with third-party applications such as
billing or customer service.
♦ Performance Manager — collects and presents statistics for performance analysis. It
collects metrics for Windows, UNIX, and MVS hosts, file systems, Oracle
databases, Fibre Channel switches, and Symmetrix and CLARiiON storage arrays.
Performance Manager also allows users to generate web-based reports on scheduled
intervals; hourly, daily, weekly.
♦ Symmetrix Manager — monitors, automates operations, and provisions storage for
Symmetrix arrays.
♦ Symmetrix Optimizer — automatically tunes Symmetrix arrays for optimal
performance based on business requirements.
♦ SRDF-TimeFinder Manager — monitoring, provisioning, and automation of SRDF
and TimeFinder in the storage environment.
♦ Common Array Manager — simplifies the management of multivendor storage
environments by providing monitoring and reporting capabilities for HP
StorageWorks, Hitachi/HP/Sun, and IBM ESS storage arrays.
♦ SAN Architect — an online, subscription-based product that provides users with an
easy-to-use SAN design, validation, and modeling solution. Users can model
changes to existing SANs or design new topologies with immediate validation
against EMC internal knowledge bases. SAN Architect guides users through the
entire design process and automatically validates array, switch, and topology
choices.
♦ CLARiiON, Celerra, and Network Appliance management — ControlCenter can
also be used with other storage element managers like Navisphere® Manager,
Connectrix Manager, and non-EMC array and switch managers. Many of these can
be launched from the console to deliver extended functionality or capabilities.

2.8 EMC PowerPath
EMC PowerPath is host-based software that works with networked storage systems to
intelligently manage I/O paths. PowerPath manages multiple paths to a storage array.
Supporting multiple paths enables recovery from path failure because PowerPath
automatically detects path failures and redirects I/O to other available paths. PowerPath
also uses sophisticated algorithms to provide dynamic load balancing for several kinds
of path management policies that the user can set. With the help of PowerPath, systems
administrators are able to ensure that applications on the host have highly available
access to storage and perform optimally at all times.
A key feature of path management in PowerPath is dynamic, multipath load balancing.
Without PowerPath, an administrator must statically load balance paths to logical
devices to improve performance. For example, based on current usage, the administrator
might configure three heavily used logical devices on one path, seven moderately used
logical devices on a second path, and 20 lightly used logical devices on a third path. As

2-38

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

I/O patterns change, these statically configured paths may become unbalanced, causing
performance to suffer. The administrator must then reconfigure the paths, and continue
to reconfigure them as I/O traffic between the host and the storage system shifts in
response to use changes.
Designed to use all paths concurrently, PowerPath distributes I/O requests to a logical
device across all available paths, rather than requiring a single path to bear the entire I/O
burden. PowerPath can distribute the I/O for all logical devices over all paths shared by
those logical devices, so that all paths are equally burdened. PowerPath load balances
I/O on a host-by-host basis, and maintains statistics on all I/O for all paths. For each I/O
request, PowerPath intelligently chooses the least-burdened available path, depending on
the load-balancing and failover policy in effect. In addition to improving I/O
performance, dynamic load balancing reduces management time and downtime because
administrators no longer need to manage paths across logical devices. With PowerPath,
configurations of paths and policies for an individual device can be changed
dynamically, taking effect immediately, without any disruption to the applications.
PowerPath provides the following features and benefits:
♦ Multiple paths, for higher availability and performance — PowerPath supports
multiple paths between a logical device and a host bus adapter (HBA, a device
through which a host can issue I/O requests). Having multiple paths enables the host
to access a logical device even if a specific path is unavailable. Also, multiple paths
can share the I/O workload to a given logical device.
♦ Dynamic multipath load balancing — Through continuous I/O balancing, PowerPath
improves a host’s ability to manage heavy I/O loads. PowerPath dynamically tunes
paths for performance as workloads change, eliminating the need for repeated static
reconfigurations.
♦ Proactive I/O path testing and automatic path recovery — PowerPath periodically
tests failed paths to determine if they are available. A path is restored automatically
when available, and PowerPath resumes sending I/O to it. PowerPath also
periodically tests available but unused paths, to ensure they are operational.
♦ Automatic path failover — PowerPath automatically redirects data from a failed I/O
path to an alternate path. This eliminates application downtime; failovers are
transparent and non-disruptive to applications.
♦ Enhanced high-availability cluster support — PowerPath is particularly beneficial in
cluster environments, as it can prevent interruptions to operations and costly
downtime. PowerPath’s path failover capability avoids node failover, maintaining
uninterrupted application support on the active node in the event of a path
disconnect (as long as another path is available).
♦ Consistent split — PowerPath allows users to perform TimeFinder consistent splits
by suspending device writes at the host level for a fraction of a second while the
foreground split occurs. PowerPath software provides suspend-and-resume
capability that avoids inconsistencies and restart problems that can occur if a
database-related BCV is split without first quiescing the database.
♦ Consistency Groups — Consistency Groups are a composite group of Symmetrix
devices specially configured to act in unison to maintain the integrity of a database

EMC PowerPath

2-39

EMC Foundation Products

distributed across multiple SRDF units controlled by an open systems host
computer.

2.9 EMC Replication Manager
EMC Replication Manager is an EMC software application that dramatically simplifies
the management and use of disk-based replications to improve the availability of user’s
mission-critical data and rapid recovery of that data in case of corruption.
Replication Manager helps users manage replicas as if they were tape cartridges in a tape
library unit. Replicas may be scheduled or created on demand, with predefined
expiration periods and automatic mounting to alternate hosts for backups or scripted
processing. Individual users with different levels of access ensure system and replica
integrity. In addition to these features, Replication Manager is fully integrated with
many critical applications such as DB2 LUW, Oracle, and Microsoft Exchange.
Replication Manager makes it easy to create point-in-time, disk-based replicas of
applications, file systems, or logical volumes residing on existing storage arrays. It can
create replicas of information stored in the following environments:
♦ Oracle databases
♦ DB2 LUW databases
♦ Microsoft SQL Server databases
♦ Microsoft Exchange databases
♦ UNIX file systems
♦ Windows file systems
The software utilizes a Java-based client-server architecture. Replication Manager can:
♦ Create point-in-time replicas of production data in seconds.
♦ Facilitate quick, frequent, and non-destructive backups from replicas.
♦ Mount replicas to alternate hosts to facilitate offline processing (for example,
decision-support services, integrity checking, and offline reporting).
♦ Restore deleted or damaged information quickly and easily from a disk replica.
♦ Set the retention period for replicas so that storage is made available automatically.
Replication Manager has a generic storage technology interface that allows it to connect
and invoke replication methodologies available on:
♦ EMC Symmetrix arrays
♦ EMC CLARiiON arrays
♦ HP StorageWorks arrays

2-40

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

EMC Foundation Products

Replication Manager uses Symmetrix API (SYMAPI) Solutions Enabler software and
interfaces to the storage array’s native software to manipulate the supported disk arrays.
Replication Manager automatically controls the complexities associated with creating,
mounting, restoring, and expiring replicas of data. Replication Manager performs all of
these tasks and offers a logical view of the production data and corresponding replicas.
Replicas are managed and controlled with the easy-to-use Replication Manager console.

EMC Replication Manager

2-41

EMC Foundation Products

2-42

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 3

Creating Oracle
Database Clones

This chapter presents these topics:
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
3.10

Overview ............................................................................................................ 3-2
Comparing recoverable and restartable copies of databases .............................. 3-2
Copying the database with Oracle shutdown ..................................................... 3-3
Copying a running database using EMC consistency technology...................... 3-8
Copying the database with Oracle in hot backup mode ................................... 3-13
Replicating Oracle using Replication Manager................................................ 3-20
Transitioning disk copies to Oracle database clones ........................................ 3-22
Oracle transportable tablespaces ...................................................................... 3-28
Cross-platform transportable tablespaces......................................................... 3-33
Choosing a database cloning methodology ...................................................... 3-36

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

3-1

Creating Oracle Database Clones

This section describes the Oracle database cloning process using various EMC products.
Determining which replication products to use depends on the customer’s requirements
and database environment. Products such as TimeFinder and Replication Manager
provide an easy method copying of Oracle databases in a single Symmetrix array.
This section describes the database cloning process. A database cloning process
typically includes some or all of the following steps, depending on the copying
mechanism selected and the desired usage of the database clone:
♦ Preparing the array for replication
♦ Conditioning the source database
♦ Making a copy of the database volumes
♦ Resetting the source database
♦ Presenting the target database copy to a server
♦ Conditioning the target database copy

3.1 Overview
There are many choices when cloning databases with EMC array-based replication
software. Each software product has differing characteristics that affect the final
deployment. A thorough understanding of the options available leads to an optimal
replication choice.
An Oracle database can be in one of three data states when it is being copied:
♦ Shutdown
♦ Processing normally
♦ Conditioned using hot-backup mode
Depending on the data state of the database at the time it is copied, the database copy
may be restartable or recoverable. This section begins with a discussion of recoverable
and restartable database clones. It then describes various approaches to data replication
using EMC software products and how the replication techniques is used in combination
with the different database data states to facilitate the database cloning process.
Following that, database clone usage considerations are discussed along with
descriptions of the procedures used to deploy database clones across various operatingsystems platforms.

3.2 Comparing recoverable and restartable copies of databases
The Symmetrix-based replication technologies described in this section can create two
types of database copies: recoverable or restartable. A significant amount of confusion
exists between these two types of database copies; a clear understanding of the
differences between the two is critical to ensure the appropriate application of each
method when a cloned Oracle environment is required.

3-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

3.2.1 Recoverable disk copies
A recoverable database copy is one in which logs can be applied to the database data
state and the database is rolled forward to a point in time after the database copy is
created. A recoverable Oracle database copy is intuitively easy for DBAs to understand
since maintaining recoverable copies, in the form of backups, is an important DBA
function. In the event of a failure of the production database, the ability to recover the
database not only to the point in time when the last backup was taken, but also to roll
forward subsequent transactions up to the point of failure, is a key feature of the Oracle
database.

3.2.2 Restartable disk copies
If a copy of a running Oracle system is created using EMC consistency technology
without putting the database in hot backup mode, the copy is a DBMS restartable copy.
This means that when the DBMS is started on the restartable copy, it performs crash
recovery. First, all transactions recorded as committed and written to the redo log, but
which may not have had corresponding data pages written to the data files are rolled
forward using the redo logs. Second, after the application of log information completes,
Oracle rolls back any changes that were written to the database (dirty pages flushed to
disk for example), but were never actually committed by a transaction. The state
attained is often referred to as a transactionally consistent point in time. It is essentially
the same process that the RDBMS would undergo if the server suffered an unanticipated
interruption such as a power failure.
Roll-forward recovery using archive logs to a point in time after the disk copy is created
is unsupported on an Oracle restartable database copy.

3.3 Copying the database with Oracle shutdown
Ideally, a copy of an Oracle database should be taken while the database is shut down.
Taking a copy after the database has been shut down normally ensures a clean copy for
backups to tape or for fast startup of the cloned database. In addition, a cold copy of a
database is in a known transactional data state which, for some application requirements,
is exceedingly important. Copies of running databases are in unknown transactional
data states.
While a normal shutdown is desirable, it is not always feasible with an active Oracle
database. In many cases, applications and databases must be forced to completely shut
down. Rarely, the shutdown abort command may be required to successfully shut
down the database. For any abnormal shutdowns, it is recommended that the database
be restarted allowing recovery and cleanup of the database, and then be shut down
normally. This ensures a clean, consistent copy of the database is available for the copy
procedure.
One primary method of creating copies of an Oracle database is through the use of the
EMC local replication product, TimeFinder. TimeFinder is also used by Replication
Manager to make database copies. Replication Manager facilitates the automation and
management of database clones.

Copying the database with Oracle shutdown

3-3

Creating Oracle Database Clones

TimeFinder comes in three different forms, TimeFinder/Mirror, TimeFinder/Clone and
TimeFinder/Snap. These were discussed in general terms in Chapter 2. Here, they are
used in a database context.

3.3.1 Creating Oracle copies using TimeFinder/Mirror
TimeFinder/Mirror is an EMC software product that allows an additional hardware
mirror to be attached to a source volume. The additional mirror is a specially designated
volume in the Symmetrix configuration called a business continuance volume (BCV).
The BCV is synchronized to the source volume through a process called an establish.
While the BCV is established, it is not ready to all hosts. At an appropriate time, the
BCV can be split from the source volume to create a complete point-in-time copy of the
source data that can be used for multiple different purposes including backup, decision
support, regression testing, and such.
Groups of BCVs are managed together using SYMCLI device or composite groups.
Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Mirror operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-1 shows how to use TimeFinder/Mirror to make a database copy of a cold
Oracle database.

Figure 3-1

Copying a cold (shutdown) Oracle database with TimeFinder/Mirror

1. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g device_group establish –full -noprompt

3-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Note that the first iteration of the establish needs to be a full synchronization.
Subsequent iterations by default are incremental if the –full keyword is omitted.
Once the command is issued, the array begins the synchronization process using
only Symmetrix resources. Since this operation occurs independently from the host,
the process must be interrogated to see when it completes. The command to
interrogate the synchronization process is:
symmir –g device_group verify

This command will return a 0 return code when the synchronization operation is
complete. Alternatively, synchronization can be verified using the following:
symmir -g device_group query

After the volumes are synchronized, the split command can be issued at any time.
2. Once BCV synchronization is complete, bring down the database to make a copy of
a cold database. Execute the following Oracle commands:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

3. When the database is deactivated, split the BCV mirrors using the following
command:
symmir –g device_group split -noprompt

The split command takes a few seconds to process. The database copy on the BCVs
is now ready for further processing.
4. The source database can now be activated and made available to users once again.
SQL> startup;

3.3.2 Creating Oracle copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume must be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-2 on page 3-6 shows how to use TimeFinder/Clone to make a copy of a cold
Oracle database onto BCV devices.

Copying the database with Oracle shutdown

3-5

Creating Oracle Database Clones

Figure 3-2

Copying a cold Oracle database with TimeFinder/Clone

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g device_group create -noprompt

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior synchronization of data is necessary. After the
TimeFinder/Clone session is created, it can be activated consistently.
2. Once the create commandis complete, shut down the database to make a cold disk
copy of the database. Execute the following Oracle commands:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

3. With the database down, activate the TimeFinder/Clone:
symclone –g device_group activate -noprompt

After an activate command, the database copy provided by TimeFinder/Clone is
immediately available for further processing even though the copying of data may
not have completed.
4. Activate the source database to make available to users once again:
SQL> startup;

3-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Databases copied using TimeFinder/Clone are subject to Copy on First Write (COFW)
and Copy On Access (COA) penalties. The COFW penalty means that if a track is
written to the source volume and it not copied to the target volume, it must first be
copied to the target volume before the write from the host is acknowledged. COA
means that if a track on a TimeFinder/Clone volume is accessed before it is copied, it
must first be copied from the source volume to the target volume. This causes additional
disk read activity to the source volumes and could be a source of disk contention on busy
systems.

3.3.3 Creating Oracle copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving, pointerbased copies of disk volumes using VDEVs and SAVDEVs. The VDEVs contain
pointers either to the source data (when it is unchanged) or to the SAVDEVs (when the
data has changed).
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix A provides examples of these commands.
Figure 3-3 shows how to use TimeFinder/Snap to make a copy of a cold Oracle
database.

Figure 3-3

Copying a cold Oracle database with TimeFinder/Snap

Copying the database with Oracle shutdown

3-7

Creating Oracle Database Clones

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g device_group create -noprompt

2. Once the create operation has completed, shut down the database to make a cold
TimeFinder/Snap of the DBMS. Execute the following Oracle commands:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

3. With the database down, the TimeFinder/Snap copy can now be activated:
symsnap –g device_group activate -noprompt

After activating the snap, the pointer-based database copy on the VDEVs is
available for further processing.
4. The source database can be started again. Use the following Oracle command:
SQL> startup;

Databases copied using TimeFinder/Snap are subject to a COFW penalty while the snap
is activated. The COFW penalty means that if a track is written to the source volume
and it has not been copied to the snap-save area, it must first be copied to the save area
before the write from the host is acknowledged.

3.4 Copying a running database using EMC consistency technology
The replication of a running database system involves a database copying technique that
is employed while the database is servicing applications and users. The database
copying technique uses EMC consistency technology combined with an appropriate data
copy process like TimeFinder/Mirror, TimeFinder/Clone, or such. TimeFinder/CG
allows for the running database copy to be created in an instant through use of the
-consistent key word on the split or activate commands. The image created in this
way is in a dependent-write consistent data state and is used as a restartable copy of the
database.
Databases management systems enforce a principle of dependent-write I/O. That is, no
dependent-write I/O will be issued until the predecessor write that it is dependent on has
completed. This type of programming discipline is used to coordinate database and log
updates within a database management system and allows those systems to be restartable
in event of a power failure. Dependent-write consistent data states are created when
database management systems are exposed to power failures. Using EMC consistency
technology options during the database cloning process also creates a database copy that
has a dependent-write-consistent data state. Chapter 2 provides more information on
EMC consistency technology.
Oracle can be copied while it is running and processing transactions. The following
sections describe how to copy a running Oracle database using TimeFinder technology.

3-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

3.4.1 Creating Oracle copies using TimeFinder/Mirror
TimeFinder/Mirror is an EMC software product that allows an additional hardware
mirror to be attached to a source volume. The additional mirror is a specially designated
volume in the Symmetrix configuration called a BCV. The BCV is synchronized to the
source volume through a process called an establish. While the BCV is established, it is
not ready to all hosts. At an appropriate time, the BCV can be split from the source
volume to create a complete point-in-time copy of the source data that can be used for
multiple different purposes including backup, decision support, regression testing, and
such.
Groups of BCVs are managed together using SYMCLI device or composite groups.
Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Mirror operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-4 shows how to use TimeFinder/Mirror and EMC consistency technology to
make a database copy of a running Oracle database.

Figure 3-4

Copying a running Oracle database with TimeFinder/Mirror

5. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g device_group establish –full -noprompt

Note that the first iteration of the establish needs to be a full synchronization.
Subsequent iterations are incremental and do not need the -full keyword. Once the
command is issued, the array begins the synchronization process using only
Symmetrix resources. Since this operation occurs independently from the host, the

Copying a running database using EMC consistency technology

3-9

Creating Oracle Database Clones

process must be interrogated to see when it completes. The command to interrogate
the synchronization process is:
symmir –g device_group verify

This command will return a 0 return code when the synchronization operation is
complete. Alternatively, verify synchronization using the following:
symmir -g device_group query

6. When the volumes are synchronized, issue the split command:
symmir –g device_group split –consistent -noprompt

The –consistent keyword tells the Symmetrix unit to use ECA (Enginuity
Consistency Assist) to momentarily suspend writes to the disks while the split is
being processed. The effect of this is to create a point-in-time copy of the database
on the BCVs. It is similar to the image created when there is a power outage that
causes the server to crash. This image is a restartable copy. The database copy on
the BCVs is then available for further processing.
Since there was no specific coordination between the database state and the execution of
the consistent split, the copy is taken independent of the database activity. In this way,
EMC consistency technology can be used to make point-in-time copies of multiple
systems atomically, resulting in a consistent point-in-time with respect to all applications
and databases included in the consistent split.

3.4.2 Creating Oracle copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume must be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-5 on page 3-11 shows how to use TimeFinder/Clone to make a copy of a
running Oracle database onto BCV devices.

3-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Figure 3-5

Copying a running Oracle database with TimeFinder/Clone

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g device_group create -noprompt

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior copying of data is necessary.
2. After the TimeFinder/Clone relationship is created, activate it consistently:
symclone –g device_group activate –consistent -noprompt

The –consistent keyword tells the Symmetrix to use ECA to momentarily suspend
writes to the source disks while the TimeFinder/Clone is being activated. The effect
of this is to create a point-in-time copy of the database on the target volumes. It is a
copy similar in state to that created when there is a power outage resulting in a
server crash. This copy is a restartable copy. After the activate command, the
database copy on the TimeFinder/Clone devices is available for further processing.
Since there was no specific coordination between the database state and the execution of
the consistent split, the copy is taken independent of the database activity. In this way,
EMC consistency technology can be used to make point-in-time copies of multiple
systems atomically, resulting in a consistent point-in-time with respect to all applications
and databases included in the consistent split.
Databases copied using TimeFinder/Clone are subject to COFW and COA penalties.
The COFW penalty means that the first time a track is written to the source volume and
it has not been copied to the target volume, it must first be copied to the target volume

Copying a running database using EMC consistency technology

3-11

Creating Oracle Database Clones

before the write from the host is acknowledged. Subsequent writes to tracks that have
already been copied do not suffer from the penalty. COA means that if a track on a
target volume is accessed before it has been copied, it must first be copied from the
source volume to the target volume. This causes additional disk read activity to the
source volumes and could be a source of disk contention on busy systems.

3.4.3 Creating Oracle copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving, pointerbased copies of disk volumes using Virtual Devices (VDEVs) and save devices
(SAVDEVs). The VDEVs contain pointers either to the source data (when it is
unchanged) or to the SAVDEVs (when the data has changed).
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix A provides examples of these commands.
Figure 3-6 shows how to use TimeFinder/Snap to make a copy of a running Oracle
database.

Figure 3-6

3-12

Copying a running Oracle database with TimeFinder/Snap

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g device_group create -noprompt

After the TimeFinder/Snap is created, all pointers from the VDEVs are directed at
the source volumes. No data has been copied at this point. The snap can be
activated consistently using the consistent activate command.
2. Once the create operation has completed, execute the activate command can with
the –consistent option to perform the consistent snap:
symsnap –g device_group activate –consistent -noprompt

The –consistent keyword tells the Symmetrix to use ECA to momentarily suspend
writes to the disks while the activate command is being processed. The effect of
this is to create a point-in-time copy of the database on the VDEVs. It is similar to
the state created when there is a power outage that causes the server to crash. This
image is a restartable copy. The database copy on the VDEVs is available for
further processing.
Since there was no specific coordination between the database state and the execution of
the consistent split, the copy is taken independent of the database activity. In this way,
EMC consistency technology can be used to make point-in-time copies of multiple
systems atomically, resulting in a consistent point in time with respect to all applications
and databases included in the consistent split.
Databases copied using TimeFinder/Snap are subject to COFW penalty while the snap is
activated. The COFW penalty means that if a track is written to the source volume and
it has not been copied to the snap-save area, it must first be copied to the snap-save area
before the write from the host is acknowledged.

3.5 Copying the database with Oracle in hot backup mode
For many years, Oracle has supported hot backup mode, which provides the capability to
use split-mirroring technology while the database is online and create a recoverable
database on the copied devices. During this process, the database is fully available for
reads and writes. However, instead of writing change vectors (such as the rowid, before,
and after images of the data) to the online redo log, entire blocks of data are written.
These data blocks are then used to overwrite any potential inconsistencies in the data
files. While this enables the database to recover itself and create a consistent point-intime image after recovery, it also degrades performance while the database is in hot
backup mode.
An important consideration when using hot backup mode to create a copy of the
database is the need to split the archive logs separately from the database. This is
because Oracle must recover itself to the point after all of the tablespaces are taken out
of hot backup mode. If the hypervolumes containing the archive logs are split at the
same time as the data volumes, the marker indicating the tablespaces are out of hot
backup mode will not be found in the last archive log. As such, the archive logs must be

Copying the database with Oracle in hot backup mode

3-13

Creating Oracle Database Clones

split after the database is taken out of hot backup mode, so the archive log devices (and
generally the redo logs as well) must be separate from the other data files.
The following sections describe the steps needed to put tablespaces or the entire
database into hot backup mode and take it out again. Appendix D provides a sample
script showing how hot backup mode is used to create a recoverable Oracle database
image.

3.5.1 Putting the tablespaces or database into hot backup mode
To create a consistent image of Oracle while in hot backup mode, each of the
tablespaces in the database must be put into hot backup mode before copying can be
performed. The following command connects to the database instance and issues the
commands to put the tablespaces (in this case, SYSTEM, DATA, and INDEXES) into
hot backup mode:
sqlplus “/
SQL> alter
SQL> alter
SQL> alter
SQL> alter

as sysdba”
system archive log current;
tablespace DATA begin backup;
tablespace INDEXES begin backup;
tablespace SYSTEM begin backup;

Alternatively, with Oracle10g, the entire database can be put into hot backup mode with:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

When these commands are issued, data blocks for the tablespaces are flushed to disk and
the deatafile headers are updated with the last SCN. Further updates of the SCN to the
datafile headers are not performed. When these files are copied, the nonupdated SCN in
the datafile headers signifies to the database that recovery is required.

3.5.2 Taking the tablespaces or database out of hot backup mode
To take the tablespaces out of hot backup mode, connect to the database and issue the
following commands:
sqlplus “/
SQL> alter
SQL> alter
SQL> alter
SQL> alter

as sysdba”
tablespace DATA end backup;
tablespace INDEXES end backup;
tablespace SYSTEM end backup;
system archive log current;

When these commands complete, the database is returned to its normal operating state.
Alternatively, with Oracle10g, the entire database can be put into hot backup mode with:
sqlplus “/ as sysdba”
SQL> alter database end backup;
SQL> alter system archive log current;

3-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

The log file switch command is used to ensure that the marker indicating that the
tablespaces have been taken out of hot backup mode is found in an archive log.

3.5.3 Creating Oracle copies using TimeFinder/Mirror
TimeFinder/Mirror is an EMC software product that allows an additional hardware
mirror to be attached to a source volume. The additional mirror is a specially designated
volume in the Symmetrix configuration, called a BCV. The BCV is synchronized to the
source volume through a process called an establish. While the BCV is established it is
not ready to all hosts. At an appropriate time, the BCV can be split from the source
volume to create a complete point-in-time copy of the source data that can be used for
multiple different purposes including backup, decision support, regression testing, etc.
Groups of BCVs are managed together using SYMCLI device or composite groups.
Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Mirror operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-7 shows how to use TimeFinder/Mirror to make a copy of an Oracle database
in hot backup mode.

Figure 3-7

Copying an Oracle database in hot backup mode with TimeFinder/Mirror

1. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g data_group establish –full -noprompt
symmir –g log_group establish –full -noprompt

Copying the database with Oracle in hot backup mode

3-15

Creating Oracle Database Clones

Note that the first iteration of the establish needs to be a “full” synchronization.
Subsequent iterations are incremental and do not need the –full keyword. Once the
command is issued, the array begins the synchronization process using only
Symmetrix resources. Since this is asynchronous to the host, the process must be
interrogated to see when it is finished. The command to interrogate the
synchronization process is:
symmir –g data_group verify
symmir –g log_group verify

This command will return a 0 return code when the synchronization operation is
complete.
2. When the volumes are synchronized, put the database in hot backup mode. Connect
to the database and issue the following commands:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute a split of the standard and BCV relationship:
symmir –g data_group split -noprompt

The –consistent keyword is not used here as consistency is being provided by the
database. The Data BCV(s) now contain an inconsistent copy of the database that
can be made consistent through recovery procedures using the archive logs. This is
a recoverable database. Usage of recoverable copies of databases is described in
Section 3.2.1.
4. After the replicating process completes, take the database (or tablespaces) out of hot
backup mode on the source database:
SQL> alter database end backup;
SQL> alter system archive log current;

5. After the tablespaces are taken out of hot backup mode and a log switch is
performed, split the Log BCV devices from their source volumes:
symmir –g log_group split -noprompt

3.5.4 Creating Oracle copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume needs to be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix B provides examples of these commands.

3-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Figure 3-8 shows how to use TimeFinder/Clone to make a copy of an Oracle database in
hot backup mode onto BCV devices.

Figure 3-8

Copying an Oracle database in hot backup mode with TimeFinder/Clone

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g data_group create -noprompt
symclone –g log_group create -noprompt

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior copying of data is necessary.
2. Place the Oracle database in hot backup mode:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute an “activate” of the TimeFinder/Clone:
symclone –g data_group activate -noprompt

The –consistent keyword is not used here as consistency is being provided by the
database. The data-clone devices now contain an inconsistent copy of the database
that can be made consistent through recovery procedures using the archive logs.
This is a recoverable database. Section 3.7.2 describes use of recoverable copies of
databases.

Copying the database with Oracle in hot backup mode

3-17

Creating Oracle Database Clones

4. After the replicating process completes, take the database (or tablespaces) out of hot
backup mode on the source database:
SQL> alter database end backup;
SQL> alter system archive log current;

5. After the tablespaces are taken out of hot backup mode and a log switch is
performed, activate the log clone devices:
symclone –g log_group activate -noprompt

Databases copied using TimeFinder/Clone are subject to COFW and COA penalties.
The COFW penalty means that the first time a track is written to the source volume and
it has not been copied to the target volume, it must first be copied to the target volume
before the write from the host is acknowledged. Subsequent writes to tracks already
copied, do not suffer from the penalty. COA means that if a track on a target volume is
accessed before it is copied, it must first be copied from the source volume to the target
volume. This causes additional disk read activity to the source volumes and could be a
source of disk contention on busy systems.

3.5.5 Creating Oracle copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving pointer-based
copies of disk volumes using VDEVs and SAVDEVs. The VDEVs contain pointers
either to the source data (when it is unchanged) or to the SAVDEVs (when the data has
changed).
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix B provides examples of these commands.
Figure 3-9 on page 3-19 shows how to use TimeFinder/Snap to make a copy of an
Oracle database in hot backup mode.

3-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Figure 3-9

Copying an Oracle database in hot backup mode with TimeFinder/Snap

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g data_group create -noprompt
symsnap –g log_group create -noprompt

Unlike TimeFinder/Mirror, the snap relationship is created and activated when it is
needed. No prior copying of data is necessary. The create operation establishes the
relationship between the standard devices and the VDEVs and it also creates the
protection metadata.
2. After the snaps are created, place the Oracle database in hot backup mode:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute an “activate” of the TimeFinder/Snap for the data devices:
symsnap –g data_group activate -noprompt

The –consistent keyword is not used here because consistency is being provided by
the database. The VDEVs (and possibly SAVDEVs) contain a pointer-based copy
of the database while it is in hot backup mode. This is a recoverable database copy.
Section 3.7.2 describes use of recoverable copies of Oracle databases.

Copying the database with Oracle in hot backup mode

3-19

Creating Oracle Database Clones

4. Once the snap activate process completes, take the database (or tablespaces) out of
hot backup mode on the source database:
SQL> alter database end backup;
SQL> alter system archive log current;

5. After the database is taken out of hot backup mode and a log switch is performed,
activate the Log snap devices:
symsnap –g log_group activate -noprompt

Databases copied using TimeFinder/Snap are subject to a COFW penalty while the snap
is activated. The COFW penalty means that if a track is written to the source volume
and it has not been copied to the snap save area, it must first be copied to the save area
before the write from the host is acknowledged.

3.6 Replicating Oracle using Replication Manager
EMC Replication Manager is used to manage and control the TimeFinder copies of an
Oracle database. The RM product has a GUI and command line and provides the
capability to:
♦ Autodiscover the standard volumes holding the database.
♦ Identify the pathname for all database files.
♦ Identify the location of the archive log directories.
♦ Identify the location of the database binaries, dump files, and such.
Using this information, RM can set up TimeFinder Groups with BCVs or VDEVs,
schedule TimeFinder operations and manage the creation of database copies, expiring
older versions as needed.
Figure 3-10 on page 3-21 demonstrates the steps performed by Replication Manager
using TimeFinder/Mirror to create a database copy to use for multiple purposes.

3-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

Figure 3-10

Using Replication Manager to make a TimeFinder copy of Oracle

Replication Manager does the following:
1. Logs in to the database and discovers the locations of all the datafiles and logs on
the Symmetrix devices. Note that the dynamic nature of this activity will handle the
situation when extra volumes are added to the database. The procedure will not
have to change.
2. Establishes the standards to the BCVs in the Symmetrix system. Replication
Manager polls the progress of the establish process until the BCVs are synchronized,
and then moves on to the next step.
3. Performs a log switch to flush changes to disk, minimizing recovery required of the
copied database.
4. Puts the Oracle database in hot backup mode, discussed in Section 3.5.1.
5. Issues a TimeFinder split, to detach the Data BCVs from the standard devices.
6. Takes the Oracle database out of hot backup mode, as discussed in Section 3.5.2.
7. Performs another log switch to flush the end of hot backup mode marker from the
online redo logs to an archive log.
8. Creates a copy of a backup control file.
9. Copies the backup control file and additional catalog information the Replication
Manager host.

Replicating Oracle using Replication Manager

3-21

Creating Oracle Database Clones

10. Copies the database archive logs to the Replication Manager host for use in the
restore process.

3.7 Transitioning disk copies to Oracle database clones
The method employed to enable a database copy for use depends on how the copy was
created. A database copy created while the source database was in hot backup mode
requires Oracle recovery before the database can be opened for normal processing. This
requires the database be started in mount mode and recovery started through the recover
database command until the point the database was taken out of hot backup mode (or
beyond if desired).
If a copy of a running database was created using EMC consistency technology without
using hot backup mode, it can be restarted only. Currently, no roll-forward log apply to
a point in time after the copy was created is supported by Oracle.
A database copy created with the EMC consistency technology should also be restarted
on another server, one different from the one that sees the source database. This is
because both the source and target databases have the datafile paths and the same
database ID, and therefore can not coexist on the same server. Oracle provides
mechanisms to change the database ID through a utility called nid. Additionally, the
paths to the datafiles can change.
The following sections describe how to restart a database copy created from a cold
database, with the database running using EMC consistency technology, and also a
database copy made while in hot backup mode. Details of how to deal with host-related
issues when processing the database copy are discussed first.

3.7.1 Host considerations
One of the primary considerations when starting a copy of an Oracle database is whether
to present it back to the same host or mount the database on another host. While it is
significantly simpler to restart a database on a secondary host, it is still possible to restart
a copy of the database on the same host with only a few extra steps. The extra steps
required to mount a database to the same host, mounting a set of copied volumes back to
the same host, changing the mount points, and relocating thedatafiles, are described next.
3.7.1.1 Mounting a set of copied volumes to the same host

Before the database can be presented back to the same host, the hypervolumes must be
presented. Additionally, operating system and logical volume specific commands must
be run to make the volumes and file systems (if applicable available). Appendix C
provides detailed procedures by operating system.
3.7.1.2 Relocating a database copy

Relocating the copy of an Oracle database is a requirement if mounting the database
back to the same server that sees the source database, or if database datafile locations
changed for whatever reason. This is accomplished by writing a backup control file to
trace and editing the file written to the $ORACLE_HOME/rdbms/log directory (by
default). Generally, the command is used to re-create a control file for the database to

3-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

use. In this case, the new control file contains a listing of new paths for the datafiles and
redo logs to point to. With the addition of an initialization parameter file, the new
database can be discovered and started for use.
The following steps are required to generate a file to use to mount a database copy on
the same or new host with new datafile locations.
1. Generate the file containing the script to re-create the control file.
sqlplus “/ as sysdba”
SQL> alter database backup controlfile to trace;

2. Find and edit the script. The trace file is written to background_dump_dest (by
default into the ORACLE_HOME/rdbms/log directory on a UNIX system) and is in
the form of SID_ora_nnnnn.trc where nnnnn is a number. It is the customer’s
responsibility to write this to a new file (create_control.sql for example) before
editing the file. Following is an example of backup control file written to trace:
Dump file /oracle/oracle9i/rdbms/log/test_ora_20748.trc
Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit
Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.7.0 – Production
ORACLE_HOME = /oracle/oracle9i
System name:
SunOS
Node name:
l82bk050
Release:
5.8
Version:
Generic_108528-29
Machine:
sun4u
Instance name: test
Redo thread mounted by this instance: 1
Oracle process number: 10
Unix process pid: 20748, image: oracle@l82bk050 (TNS V1-V3)
*** SESSION ID:(9.6785) 2005-12-11 17:04:18.454
*** 2005-12-11 17:04:18.453
# The following are current System-scope REDO Log Archival
# related parameters and can be included in the database
# initialization file.
#
# LOG_ARCHIVE_DEST=''
# LOG_ARCHIVE_DUPLEX_DEST=''
#
# LOG_ARCHIVE_FORMAT=T%TS%S.ARC
# REMOTE_ARCHIVE_ENABLE=TRUE
# LOG_ARCHIVE_START=TRUE
# LOG_ARCHIVE_MAX_PROCESSES=2
# STANDBY_FILE_MANAGEMENT=MANUAL
# STANDBY_ARCHIVE_DEST=?/dbs/arch
# FAL_CLIENT=''
# FAL_SERVER=''
#
# LOG_ARCHIVE_DEST_1='LOCATION=/oracle/archive'
# LOG_ARCHIVE_DEST_1='MANDATORY NOREOPEN NODELAY'
# LOG_ARCHIVE_DEST_1='ARCH NOAFFIRM SYNC'

Transitioning disk copies to Oracle database clones

3-23

Creating Oracle Database Clones

# LOG_ARCHIVE_DEST_1=’NOREGISTER NOALTERNATE NODEPENDENCY’
# LOG_ARCHIVE_DEST_1=’NOMAX_FAILURE NOQUOTA_SIZE NOQUOTA_USED’
# LOG_ARCHIVE_DEST_STATE_1=ENABLE
#
# Below are two sets of SQL statements, each of which creates
# a new control file and uses it to open the database. The
# first set opens the database with the NORESETLOGS option and
# should be used only if the current versions of all online
# logs are available. The second set opens the database with
# the RESETLOGS option and should be used if online logs are
# unavailable.
# The appropriate set of statements can be copied from the
# trace into a script file, edited as necessary, and executed
# when there is a need to re-create the control file.
#
#
Set #1. NORESETLOGS case
#
# The following commands will create a new control file and
# use it to open the database. Data used by the recovery
# manager will be lost. Additional logs may be required for
# media recovery of offline datafiles. Use this only if the
# current version of all online logs are available.
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "TEST" NORESETLOGS
NOARCHIVELOG
-- SET STANDBY TO MAXIMIZE PERFORMANCE
MAXLOGFILES 16
MAXLOGMEMBERS 2
MAXDATAFILES 30
MAXINSTANCES 2
MAXLOGHISTORY 224
LOGFILE
GROUP 1 (
’/oracle/oradata/test/oraredo1a.dbf’,
’/oracle/oradata/test/oraredo2a.dbf’
) SIZE 10M,
GROUP 2 (
’/oracle/oradata/test/oraredo1b.dbf’,
’/oracle/oradata/test/oraredo2b.dbf’
) SIZE 10M,
GROUP 3 (
’/oracle/oradata/test/oraredo1c.dbf’,
’/oracle/oradata/test/oraredo2c.dbf’
) SIZE 10M
-- STANDBY LOGFILE
DATAFILE
’/oracle/oradata/test/orasys.dbf’,
’/oracle/oradata/test/oraundo.dbf’,
’/oracle/oradata/test/orausers.dbf’
CHARACTER SET US7ASCII
;
# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or
# immediate.
RECOVER DATABASE
# Database can now be opened normally.
ALTER DATABASE OPEN;

3-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

# Commands to add tempfiles to temporary tablespaces.
# Online tempfiles have complete space information.
# Other tempfiles may require adjustment.
ALTER TABLESPACE TEMP_TS ADD TEMPFILE
’/oracle/oradata/test/oratest.dbf’
SIZE 524288000 REUSE AUTOEXTEND OFF;
# End of tempfile additions.
#
#
Set #2. RESETLOGS case
#
# The following commands will create a new control file and
# use it to open the database. The contents of online logs
# will be lost and all backups will be invalidated. Use this
# only if online logs are damaged.
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "TEST" RESETLOGS
NOARCHIVELOG
-- SET STANDBY TO MAXIMIZE PERFORMANCE
MAXLOGFILES 16
MAXLOGMEMBERS 2
MAXDATAFILES 30
MAXINSTANCES 2
MAXLOGHISTORY 224
LOGFILE
GROUP 1 (
’/oracle/oradata/test/oraredo1a.dbf’,
’/oracle/oradata/test/oraredo2a.dbf’
) SIZE 10M,
GROUP 2 (
’/oracle/oradata/test/oraredo1b.dbf’,
’/oracle/oradata/test/oraredo2b.dbf’
) SIZE 10M,
GROUP 3 (
’/oracle/oradata/test/oraredo1c.dbf’,
’/oracle/oradata/test/oraredo2c.dbf’
) SIZE 10M
-- STANDBY LOGFILE
DATAFILE
’/oracle/oradata/test/orasys.dbf’,
’/oracle/oradata/test/oraundo.dbf’,
’/oracle/oradata/test/orausers.dbf’
CHARACTER SET US7ASCII
;
# Recovery is required if any of the datafiles are restored
# backups, or if the last shutdown was not normal or
# immediate.
RECOVER DATABASE USING BACKUP CONTROLFILE
# Database can now be opened zeroing the online logs.
ALTER DATABASE OPEN RESETLOGS;
# Commands to add tempfiles to temporary tablespaces.
# Online tempfiles have complete space information.
# Other tempfiles may require adjustment.
ALTER TABLESPACE TEMP_TS ADD TEMPFILE
’/oracle/oradata/test/oratest.dbf’
SIZE 524288000 REUSE AUTOEXTEND OFF;

Transitioning disk copies to Oracle database clones

3-25

Creating Oracle Database Clones

# End of tempfile additions.
#

3. After deciding whether to open the database with a reset logs and editing the file
appropriately, the datafile locations can change. When run, the instance will search
in the new locations for the Oracledatafiles.
sqlplus “/ as sysdba”
SQL> @create_control

This will create the new database, relocating the datafiles into the newly specified
locations.
3.7.1.3 Changing the SID, DBNAME, and DBID

A normal part of presenting a database clone to the same or a new host is changing
identifiers associated with the database. These identifiers include:
♦ SID – System ID is used to distinguish Oracle instances on a host.
♦ DBNAME – Database Name defined in the initialization parameter at database
creation and is written to the contro lfile. It specifies the service name of the
database and should be the same as that defined in the tnsnames.ora file.
♦ DBID – Database ID, which is the internal database unique identifier.
Changing the SID is a simple procedure. It is accomplished by shutting down the
database, changing the initialization parameter ORACLE_SID, and restarting the
database. The new SID will be used to name the processes that are initiated as part of
the Oracle startup procedures. An additional step needed is to create a unique
init<SID>.ora parameter file in the ORACLE_HOME/dbs directory.
Changing the DBNAME and DBID are more complicated since they are written in the
controlfile and into the database itself. Oracle provides two utilities for changing the
DBNAME and DBID. They are the dbnewid and nid utilities. In addition, the
DBNAME can be changed by re-creating the controlfile using the procedures outlined in
Section 3.7.1.2. For changing the DBNAME, the DBID, or both, these steps are
performed after any recovery procedures required are completed by the database..

3.7.2 Enabling a cold database copy
A cold database copy is a database image taken when the copied database is shut down.
A cold database copy ensures that the database copy is consistent when it is restarted; no
crash recovery is required to make the database transactionally consistent. Restarting a
database copy taken while the database was shut down does not require any crash
recovery and as such requires minimal time to restart and open.
The following steps describe the process for restarting a cold database copy. It assumes
that either the database is being started on another host, or that the processes listed in
section 3.7.1 have completed.

3-26

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

1. Use the following SYMCLI command to verify that the appropriate hypers are
available to the host:
syminq

2. After the appropriate devices are available to the host, make the operating system
aware of the devices. In addition, import the volume or disk groups and mount any
file systems. This is operating-system dependent and is discussed in Appendix C.
3. Since the database was shut down when the copy was made, no special processing is
required to restart the database. Start the database as follows:
sqlplus “/ as sysdba”
SQL> startup;

Alternatively, if additional archive logs, redo logs, and a valid control file from the
copied database is available to roll forward the database to a specified point in time,
use the following instead:
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

3.7.3 Enabling a restartable database copy
If a restartable database copy is started on the same host, the Oracle SID and DBID must
change. If it is started on a different server, the SID and DBID can be left the same or
changed. In most cases, it is appropriate to change the SID and DBID so that
connections through Oracle Net are uniquely identified. The following steps show the
process for enabling a restartable database copy.
1. Use the following SYMCLI command to verify that the appropriate hypers are
available to the host:
syminq

2. After the appropriate devices are available to the host, make the operating system
aware of the devices. In addition, import the volume or disk and any mount file
systems. This is operating-system dependent and is discussed in Appendix C.
3. Since the database was shut down when the copy was made, no special processing is
required to restart the database. Start the database as follows:
sqlplus “/ as sysdba”
SQL> startup;

3.7.4 Enabling a hot backup database copy
A database copy made with the database in hot backup mode by definition requires
recovery before the database can be opened for user transactions. The following steps
are used to recovery and open a database copy made with the database in hot backup
mode.

Transitioning disk copies to Oracle database clones

3-27

Creating Oracle Database Clones

1. Use the following SYMCLI command to verify that the appropriate hypers are
available to the host:
syminq

2. After the appropriate devices are available to the host, make the operating system
aware of the devices. In addition, import the volume or disk groups and mount any
file systems. This is operating-system dependent and is discussed in Appendix C.
3. Since the database was shutdown when the copy was made, no special processing is
required to restart the database. The following is used to start the database:
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

After applying the required logs to make the database consistent to the point where
the database was taken out of hot backup mode, the copy can open for user
transactions.

3.8 Oracle transportable tablespaces
A number of customers require Oracle objects, specifically tablespaces, to be cloned
across Oracle instances. Reasons for cloning Oracle objects include building
development environments, providing system integration and test environments,
building read-only decision support environments, or satisfying a number of other
business-related requirements for Oracle data. Oracle provides a mechanism for moving
tablespace objects from one database to another through a method called transportable
tablespaces. In conjunction with EMC replication technologies such as TimeFinder,
Open Replicator, or SRDF, pieces of an Oracle database can be cloned and attached to
an alternate database for a variety of business applications.
The transportable tablespace feature was introduced in Oracle8i to facilitate the bulk
movement of data between two Oracle databases running the same operating system and
version of Oracle. Additionally, new functionality built into Oracle10g allows
tablespaces to be transported between different operating system platforms (such as a
Sun tablespace to a Windows environment). This new enhancement allows greater
flexibility when migrating from one operating system to another or when creating
test/development systems on lower-cost operating environments.

3.8.1 Benefits and uses of transportable tablespaces
Transportable tablespaces are moved by placing the target tablespaces into read-only
mode at the Oracle level, and then copying the associated operating system files by an
external means (such as cp, dd, ftp, and so on) to place them on the host where the target
database is located. Previous methods of transferring the data, such as export and
import, required significant time and effort to migrate the data to the new instance.
Transportable tablespaces provide a simple mechanism for tablespace copies to be
incorporated into a second database environment.
There are a variety of uses for Oracle transportable tablespaces. For example, customers
may need to access data from their OLTP database to populate their data warehouse

3-28

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

system. Transportable tablespaces provide a mechanism to migrate the required data
directly into the data warehouse environment. Another example for transportable
tablespaces is migrating periodic data (for example, monthly sales data) from high-end
mirrored storage to cheaper RAID 5 volumes as the data ages and access requirements
change. Transportable tablespaces allows customers to quickly and easily move this
data between the RAID types.

3.8.2 Implementation of transportable tablespaces with EMC TimeFinder and SRDF
When implementing Oracle transportable tablespaces on a database running on an EMC
Symmetrix system, replication software such as TimeFinder or SRDF may be used to
create a clone of the tablespace(s) to be transported to the target environment. Creating
copies of the data in this manner has the advantage that no host cycles are used in the
cloning process. Additionally, following the initial full-copy synchronization,
incremental replication, in which only changed tracks are copied, can be used when
TimeFinder or SRDF are the replication method. This significantly reduces the time
required to copy the datafiles to the target environment. Finally, this process is also
easily scripted or managed through EMC management products like Replication
Manager. The next section provides an example of transportable tablespaces with the
EMC TimeFinder software.

3.8.3 Transportable tablespace example
Before implementing transportable tablespaces, a few requirements must be addressed.
For example, the source and target databases must use the same character set and
national character set. Also, a transportable tablespace cannot be mounted to a database
containing a tablespace with that name. Any users that own objects in the tablespace
may either exist or be created in the target database; objects can be transferred to other
users if required. Additionally, starting with Oracle9i, multiple block sizes can exist in
the database. If the block size used for the tablespace is not the default size for the target
database, buffer cache of the size used by the transportable tablespace must be allocated
in the target.
The major limitation with transportable tablespaces however, is that the tablespace set
(the group of tablespaces to be migrated) must be self-contained. Self-contained means
that indexes or referential integrity constraints on any tablespace object, for example the
primary key index on a table, must be included as a part of the transportable tablespace
set. Thus, in a typical customer environment where table data and indexes are separated
into their own tablespaces, either both tablespaces must be a part of the transportable
tablespace set, or the indexes must be dropped before the transportable tablespace set is
created.
Oracle provides a procedure to verify that the transportable tablespace set is selfcontained. This procedure, called TRANSPORT_SET_CHECK, is a part of the
DBMS_TTS package. This package is created as a part of the dbms_plugts script,
which is automatically run as a part of catproc. The role
EXECUTE_CATALOG_ROLE must be assigned to any user that executes this
procedure.

Oracle transportable tablespaces

3-29

Creating Oracle Database Clones

The following is an example of the steps needed to verify that a set of tablespaces can
transport successfully. This example uses two tablespaces, DATA1 and INDEX1, and
verifies that they can be successfully transported to a target database.
1. Determine the national character set in use by the source and target databases:
SELECT *
FROM
v$nls_parameters
WHERE parameter = ‘NLS_CHARACTERSET’;
The output from this SQL command on both the source and target databases should
be identical. For example:
PARAMETER
VALUE
------------------- ------------------NLS_CHARACTERSET
WE8ISO8859P1
2. Verify that tablespaces with similar names do not already exist in the target
database:
SELECT tablespace_name
FROM
dba_tablespaces;
TABLESPACE_NAME
------------------------------SYSTEM
SYSAUX
TEMP1
UNDO1
USERS1
3. Determine the users that own objects in the tablespaces to be transported, and verify
that either they exist in the target database or create them. Note that these objects
can be transferred to another user, if required.
SELECT DISTINCT owner
FROM
dba_segments
WHERE tablespace_name IN (‘DATA1’, ‘INDEX1’);
OWNER
--------------DEV1
USER1
These owners (schemas) need to be verified on the target side:
SELECT username
FROM
dba_users;
USERNAME
--------------SYS
SYSTEM
In this case, the DEV1 user exists but the USER1 user does not. The USER1 user
must be created with the command:
CREATE USER user1

3-30

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

IDENTIFIED BY user1;
4. Verify the default block sizes in each database. If different, additional block buffer
cache must be allocated in the target database of the appropriate size. Multiple
values for database block sizes were released in Oracle9i.
SELECT name, value
FROM
v$parameter
WHERE name = ‘db_block_size’;
NAME
----------------db_block_size

VALUE
-------------------8192

5. Verify if the tablespaces in the set (DATA1 and INDEX1) are self-contained:
EXECUTE dbms_tts.transport_set_check(‘DATA1,
INDEX1’,TRUE);
SELECT *
FROM
transport_set_violations;
VIOLATIONS
-------------------------------------------------------CONSTRAINT FK_SALES_ORDER_DEPT between table DEV1.SALES
in tablespace DATA1 and table DEV2.ORDER_DEPT in
tablespace DATA2
PARTITIONED TABLE DEV1.SALES is partially contained in
the transportable set
In this example, the foreign key constraint on the DEV1.SALES table would need to
be dropped. Additionally, the partitioned table DEV1.SALES would need to be
addressed.
After determining that any issues with the self-contained tablespaces are addressed, the
tablespaces need to be put into read-only mode (or taken completely offline) so that
copies of the files can be made and presented to the target host. Additionally, metadata
concerning the tablespaces must be extracted so that the tablespaces can be successfully
“plugged into” the new environment. It should be noted that extraction of the metadata
from the source database and importing it into the target is Oracle version dependent.
Following are the steps to implement a transportable tablespace:
1. Put the two tablespaces into read-only mode:
ALTER TABLESPACE data1 READ ONLY;
ALTER TABLESPACE index1 READ ONLY;

2. Extract tablespace metadata. There are two ways to do this. The first method,
available in all Oracle versions, is by using the Oracle export utility EXP. The
second, a new feature in Oracle10g, uses the Oracle Data Pump utility.
EXP

transport_tablespace = y
tablespaces = data1, index1
triggers = y

Oracle transportable tablespaces

3-31

Creating Oracle Database Clones

constraints = y
grants = y
file = d:\oracle\exp\meta1.dmp

Alternatively, Oracle Dump Pump syntax for the metadata extract is as follows:
EXPDP system/manager
DUMPFILE = meta1.dmp
DIRECTORY = d:\oracle\exp
TRANSPORT_TABLESPACES = data1,index1

3. After successfully extracting the metadata, copy the datafile(s) associated with the
tablespaces. First, the datafiles must be identified and copied to their new location
(either on the same host or a different one). A variety of methods for copying the
datafiles are available including cp, copy, cpio, tar, or the DBMS_FILE_COPY
package. Additionally, EMC software such as TimeFinder or SRDF can be used to
clone the volumes as described in the following section. In this example,
TimeFinder is used to clone the data.
SELECT tablespace_name, file_name
FROM
dba_data_files
WHERE tablespace_name in (‘DATA1’, ‘INDEX1’);

TABLESPACE_NAME
--------------DATA1
INDEX1

FILE_NAME
-------------------------------d:\oracle\oradata\db1\data1.dbf
d:\oracle\oradata\db1\index1.dbf

In this case, both required datafiles are on the d:\ drive. This volume will be
identified and replicated via the TimeFinder software. Note that careful database
layout planning is critical when TimeFinder is used for replication. First, create a
device group for the standard device used by the d:\ drive and a BCV that will be
used for the new e:\ drive. Appendix B provides examples of creating device
groups.
4. After creating the device group, establish the BCV to the standard device:
symmir –g device_group establish –full –noprompt
symmir –g device_group verify –i 30

5. After the BCV is fully synchronized with the standard device, the devices can split
since the tablespaces on the device are in read-only mode.
symmir –g device_group split -noprompt

6. A full, track-by-track copy of the d:\ drive is now available on the BCVs. Once the
BCVs are split, they become available to the host they are presented to. These
BCVs should be mounted on another host (could be the same host) that contains the
database that the transported tablespaces will be mounted to. Appendix C provides
the steps to present devices to each of the operating system types. To verify that the
volumes are presented to the host, enter:
syminq

3-32

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Creating Oracle Database Clones

7. Once the tablespace datafiles are in place, import the metadata information into the
target database:
IMP

transport_tablespace = y
datafiles = (e:\oracle\oradata\db1\data1.dbf,
e:\oracle\oradata\db1\index1.dbf)
file = d:\oracle\exp\meta1.dmp
tablespaces = (data1,index1)
tts_owners = (dev1,dev2)

Alternatively, with Data Pump in Oracle10g:
IMPDP system/manager
DUMPFILE = meta1.dmp
DIRECTORY = d:\oracle\exp\
TRANSPORT_DATAFILES = e:\oracle\oradata\db1\data1.dbf,
e:\oracle\oradata\db1\index1.dbf

8. Put the tablespaces on the target host into read/write mode:
ALTER TABLESPACE data1 READ WRITE;
ALTER TABLESPACE index1 READ WRITE;

3.9 Cross-platform transportable tablespaces
Oracle introduced a new concept with transportable tablespaces in Oracle10g. Crossplatform transportable tablespaces are an enhancement to previous functionality that
allows a single tablespace, or a set of tablespaces, to be migrated from one operating
system to another. Previously, use of transportable tablespaces required that both the
operating system and version of Oracle be the same between target and source. If the
target database is Oracle10g, these limiting feature requirements no longer apply. Crossplatform transportable tablespaces provide customers with a significantly improved
method for migrating Oracle databases from one operating system to another.

3.9.1 Overview
Cross-platform transportable tablespaces enable data from an Oracle database running
on one operating system to be cloned and presented to another database running on a
different platform. Oracle datafiles differences, as a result of the need to run on different
operating systems, are a function of byte ordering, or”endianness,” of the files. The
endian format of the datafiles is classified as either “big endian” or “little endian” (in
“big endian,” the first byte is the most significant while in “little endian”, the first byte is
the least significant). If two operating systems both use “big endian” byte ordering, the
files can transferred between operating systems and used successfully in an Oracle
database (through a feature such as transportable tablespaces). For source and target
operating systems with different byte ordering, a process to convert the datafiles from
one “endianness” to another is required.
Oracle uses an RMAN option to convert a data file from “big endian” to “little endian”
and vice versa. First, the “endianness” of the source and target operating systems must
be identified. If different, then the datafiles are read and converted by RMAN. Upon
completion, the “endianness” of the datafiles is converted to the format needed in the

Cross-platform transportable tablespaces

3-33

Creating Oracle Database Clones

new environment. The process of converting the cloned datafiles occurs either on the
source database host before copying to the new environment or once it is received on the
target host. Other than this conversion process, the steps for cross-platform
transportable tablespaces are the same as for normal transportable tablespaces.

3.9.2 Implementing cross-platform transportable tablespaces
The following example shows the process needed to implement cross-platform
transportable tablespaces for a tablespace migrating from a Solaris (big endian format) to
a Linux host (little endian format). In this example, the tablespace OR_UFS is migrated
from a Solaris to a Linux Oracle database.
1. Verify that the tablespace, or set of tablespaces, are self-contained. This means that
objects in the tablespace set must not have associated objects (such as indexes,
materialized views, or partitioned tables) outside of the specified tablespace set.
Oracle provides the procedure TRANSPORT_SET_CHECK as a part of the Oracle
provided package DBMS_TTS. For example:
EXECUTE DBMS_TTS.TRANSPORT_SET_CHECK(‘OR_UFS’, TRUE);

Any violations of the tablespace being self-contained are written to the
TRANSPORT_SET_VIOLATIONS view, and queried using:
SELECT *
FROM TRANSPORT_SET_VIOLATIONS;

If no rows are selected, the tablespace is self-contained.
2. Determine the source and target database endian formats and determine whether
conversion is required. The first step lists the endian formats for all available
operating systems. The second shows the specific format for the database platform
in use.
SELECT
platform_id, platform_name, endian_format
FROM
v$transportable_platform
ORDER BY 1;
PLATFORM
-------1
2
3
4
5
6
7
8
9
10
11
12
13
15
16

3-34

PLATFORM_NAME
-------------------------------Solaris[tm] OE (32-bit)
Solaris[tm] OE (64-bit)
HP-UX (64-bit)
HP-UX IA (64-bit)
HP Tru64 UNIX
AIX-Based Systems (64-bit)
Microsoft Windows IA (32-bit)
Microsoft Windows IA (64-bit)
IBM zSeries Based Linux
Linux IA (32-bit)
Linux 64-bit for AMD
Microsoft Windows 64-bit for AMD
Linux 64-bit for AMD
HP Open VMS
Apple Mac OS

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

ENDIAN_FORMAT
------------Big
Big
Big
Big
Little
Big
Little
Little
Big
Little
Little
Little
Little
Little
Big

Creating Oracle Database Clones

SELECT a.platform_name, a.endian_format
FROM
v$transportable_platform a, v$database b
WHERE a.platform_name = b.platform_name;

On the Solaris host, the output from this SQL command is:
PLATFORM_NAME
----------------------------Solaris[tm] OE (32-bit)

ENDIAN_FORMAT
-------------Big

On the Linux host, this command returns:
PLATFORM_NAME
----------------------------Linux IA (32-bit)

ENDIAN_FORMAT
-------------Little

3. Either shut down the database or put the tablespace(s) into read-only mode so that a
clean copy of the datafiles that make up the tablespace set can be made. A
tablespace (in this example, OR_UFS) is put into read-only mode with the
following:
SQL> ALTER TABLESPACE or_ufs READ ONLY;

4. Metadata of the tablespace set must be created and copied to the target environment.
Use either the Oracle export utility or the new Data Pump facility to create this file.
The following shows the commands to create the tablespace metadata information
using Oracle Data Pump:
expdp system/manager
dumpfile=or_ufs.dmp
directory=dpump_dir
transport_tablespaces=or_ufs
transport_full_check=Y

5. After putting the tablespace in read-only mode, the datafiles can be copied and
presented to the target host. There are many ways to manage this replication process
including host-based (cp, rcp, ftp, and so on) and storage-based methods
(TimeFinder, SRDF, Open Replicator). These new target volumes are then
presented to the target host.
6. The endianness of the data may be converted either on the storage or the target host.
In this example, the conversion process is performed after migrating the data to the
target. The Oracle RMAN utility is used to convert the data file. The following
shows an example of the RMAN conversion process:
RMAN> CONVERT DATAFILE “/ufs/or_ufs.dbf”
2> TO PLATFORM=”Linux IA (32-bit)”
3> FROM PLATFORM=”Solaris[tm] OE (64-bit)”
4> DB_FILE_NAME_CONVERT=”/ufs”,”/ufs2”
5> PARALLELISM=2;

The datafile is converted to little endian and is written to the new directory location
/ufs2 from the /ufs directory using the same filename.

Cross-platform transportable tablespaces

3-35

Creating Oracle Database Clones

7. After converting the file, the tablespace may now be “plugged” into the target
database. The Data Pump utility is used to facilitate the process.
impdp system/manager
dumpfile=or_ufs.dmp
directory=dpump_dir
transport_datafiles=/ufs2/or_ufs.dbf

3.10 Choosing a database cloning methodology
The replication technologies described in the prior sections each have pros and regarding
their applicability to solve a given business problem. Table 3-1 compares the different
methods to use and the differing attributes of those methods.
Table 3-1

Comparison of database cloning technologies
TimeFinder/Snap

TimeFinder/Clone

TimeFinder/Mirror

Replication
Manager

Maximum
number of
copies

15

Incremental: 16

Incremental: 16

Incremental: 16

Non-inc: Unlimited

Non-inc: Unlimited

Non-inc:
Unlimited

No.
simultaneous
Copies

15

16

2

2

Production
impact

COFW

COFW & COA

None

None

Scripting

Required

Required

Required

Automated

Database clone
needed a long
time

Not
recommended

Recommended

Recommended

Recommended

High write
usage to DB
clone

Not
recommended

Recommended

Recommended

Recommended

Following are some examples of the choices you might make for database cloning based
on the information in Table 3-1.
Table 3-2

3-36

Database cloning requirements and solutions
System requirements

Replication choices

The application on the source volumes is very performancesensitive and the slightest degradation will cause responsiveness
of the system to miss SLAs.

TimeFinder/Mirror
Replication
Manager

Space and economy are a real concern. Multiple copies are
needed and retained only a short period of time, with performance
not critical.

TimeFinder/Snap
Replication
Manager

More than two simultaneous copies need to be made. The copies
will live for up to a month’s time.

TimeFinder/Clone

Multiple copies are being made, some with production mount.
The copies are reused in a cycle expiring the oldest one first.

Replication
Manager

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 4

Backing Up Oracle
Environments

This chapter presents these topics:
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
4.10
4.11

Comparing recoverable and restartable copies of databases .............................. 4-2
Database organization to facilitate recovery ...................................................... 4-3
Oracle backup overview ..................................................................................... 4-4
Using EMC replication in the Oracle backup process........................................ 4-8
Copying the database with Oracle shutdown ..................................................... 4-9
Copying a running database using EMC consistency technology.................... 4-14
Copying the database with Oracle in hot backup mode ................................... 4-19
Backing up the database copy .......................................................................... 4-26
Backups using EMC Replication Manager for Oracle backups ....................... 4-26
Backups using Oracle Recovery Manager (RMAN)........................................ 4-28
Backups using TimeFinder and Oracle RMAN ............................................... 4-29

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

4-1

Backing Up Oracle Environments

As a part of normal day-to-day operations, the DBA creates backup procedures that run
one or more times a day to protect the database against errors. Errors can originate from
many sources (such as software, hardware, user, and so on) and it is the responsibility of
the DBA to provide error recovery strategies that can recover the database to a point of
consistency and also minimize the loss of transactional data. Ideally, this backup
process should be simple, efficient, and fast.
Today, the DBA is challenged to design a backup (and recovery) strategy to meet the
ever-increasing demands for availability that can also manage extremely large databases
efficiently while minimizing the burden on servers, backup systems, and operations
staff.
This section describes how the DBA can leverage EMC technology in a backup strategy
to:
♦ Reduce production impact of performing backups.
♦ Create consistent point-in-time backup images.
♦ Create restartable or recoverable database backup images.
♦ Enhance Oracle’s RMAN backup utility.
Before covering these capabilities, it is necessary to review some terminology and also
to look at best practices for Oracle database layouts that can facilitate and enhance the
backup and restore process.

4.1 Comparing recoverable and restartable copies of databases
The Symmetrix-based replication technologies for backup described in this section can
create two types of database copies: recoverable or restartable. A significant amount of
confusion exists between these two types of database images; a clear understanding of
the differences between the two is critical to ensure that the recovery goals for a
database can be met.

4.1.1 Recoverable disk copies
A recoverable database copy is a disk copy of the database in which transaction logs can
be applied to datafiles to roll forward the database content to a point in time after the
copy is created. A recoverable Oracle database copy is intuitively easy for DBAs to
understand since maintaining recoverable copies, in the form of backups, is an important
DBA function. In the event of a failure of the production database, the ability to recover
the database not only to the point in time when the last backup was taken, but also to
roll forward subsequent transactions up to the point of failure, is a key feature of the
Oracle database.
Creating recoverable images of Oracle databases with EMC replication technology
requires that the database be shut down when it is copied or if a running database is to be
replicated, the database must be in hot backup mode. This mode is described in detail
Section 4.7.1.

4-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

4.1.2 Restartable disk copies
If a copy of a running Oracle database is created using EMC consistency technology
without putting the database in hot backup mode, the copy is a DBMS restartable copy.
This means that when the restartable database copy is first brought up, it performs crash
recovery. First, all transactions recorded as committed and written to the redo log, but
which may not have had corresponding data pages written to the datafiles, are rolled
forward using the redo logs. Second, after the application of log information completes,
Oracle rolls back any changes that were written to the database (dirty block buffers
flushed to disk for example) but that were never actually committed by a transaction.
The state attained as a result of these two actions is often referred to as a transactionally
consistent point-in-time database state.
Roll-forward recovery using archive logs to a point in time after the disk copy is created
is not supported on an Oracle restartable database copy.

4.2 Database organization to facilitate recovery
It is advantageous to organize the database on disk to facilitate recovery. Since array
replication techniques copy volumes at the physical disk level (as seen by the host), all
datafiles for a database should be created on a set of disks dedicated to the database and
should not be shared with other applications and databases. For UNIX systems using a
logical volume manager (LVM), ensure that the data files reside in a volume group
dedicated to the database. Sharing with other applications can cause unnecessary work
for the array and waste space on the target volumes.
In addition to isolating the database to be copied onto its own dedicated volumes, the
database should also be divided into two parts, the data structures and the recovery
structures. The recovery structures consist of the redo logs, the archive logs, and the
control files. The database data volumes hold the data files such as the SYSTEM,
SYSAUX, and other database tablespaces. Figure 4-1 on page 4-4 depicts a TimeFinder
setup where the data structures and recovery structures have been separated onto their
own volumes.

Database organization to facilitate recovery

4-3

Backing Up Oracle Environments

Figure 4-1

Database organization to facilitate recovery

The strategy of separating data structures and recovery structures allows just the data
structures to be restored from a database disk copy or from a tape dump of the database
copy. Once the data structures are restored, the database can be rolled forward to a
designated point in time. If the data and recovery structures are not separated, the
resulting database restore will return the data state to that of the restored database image,
and no roll-forward processing will be possible as the more current logs will be
overwritten by the restore process.

4.3 Oracle backup overview
Oracle has two primary backup methods: user-managed and Recovery Manager
(RMAN). A traditional user-managed backup involves putting the database into an
allowed backup state, and then copying datafiles to tape or disk using operating-system
commands. User-managed backups require careful consideration by the DBA to ensure
that the appropriate files (archive logs, control files, datafiles, and so on) are successfully
backed up and accessible in the event a recovery process is required. Recovery typically
involves using the appropriate server platform means to perform database file restore
from backup file images, and then explicitly performing log recovery against the
restored database file images.
An alternative to user-managed backup is RMAN. RMAN provides both a utility for
easily managing the backup process and facilitating restore and recovery procedures
should they be needed. RMAN does this by providing management functions to locate
datafile members and scripting procedures to automate the backup process. It also
maintains an internal list of backup filenames and their locations to automate the
recovery process. RMAN (when not used with proxy-copy) also checks the Oracle
block integrity during the backup and logs errors if a corrupt block is found. RMAN also

4-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

allows finer recovery granularity. RMAN provides an automated, efficient utility that
simplifies the Oracle backup and recovery process.
Although it is common to see RMAN back up the production database directly, it is
recommended to offload RMAN backup to a replica of the database.

The following sections describe considerations for the various user-managed backups.
These include:
♦ Online versus offline backups – Occasionally, backups are performed by shutting
down the primary database, copying the data files to tape or disk, and then
restarting the database. However, this requires significant downtime. Given the
24x7x365 uptime requirements generally needed by IT today, hot backup mode
with Oracle database backups is widely used.
♦ Point-in-time versus roll-forward recovery backups – Historically, backup
procedures involved creating a recoverable database so that further transactions
found in the archive logs could be used to recover a database to the point in time of
a failure. However, this recovery process can be operationally complex in
federated environments.
♦ Partial (tablespace or datafile) versus entire database backups – Oracle provides the
ability to back up a single data file or tablespace in addition to backing up the entire
database. This option is useful, for example, when a tablespace will not be
accepting new transactions and is converted to read-only. After doing a final
backup, there is no reason to continue backups to this tablespace since the data is
unchanging.
♦ Incremental versus full database backups – RMAN provides the means of only
backing up changed data blocks, rather than backing up a full copy of thedatafiles.
The backup process consists of three primary steps:
♦ Preparing the database for backup (shutting down the database, putting the database
in hot backup mode, or not conditioning the database at all)
♦ Initiating the backup process to disk or tape from the operating-system or array
level
♦ Verifying the backup has completed successfully, that the backup media (tape or
disk) is protected and available for recovery (if needed), and that the backup can be
used for recovery purposes
Most backup procedures require that the database be conditioned in some way before the
data files that compose the database are backed up. This includes either shutting down
the database or putting the database in hot backup mode. However, restartable backup
images captured without requiring any special operational preparation to “condition” the
database are growing in popularity due to federated database and application
environments. Although these types of backups do not currently allow roll-forward
operations to the point of a failure as in the case of a recoverable image, they are
important particularly when used as offsite backups for DR.

Oracle backup overview

4-5

Backing Up Oracle Environments

User-managed backups require either host- or array-based replication mechanisms for
copying the data. Traditionally, backups are written to tape; although there is increasing
interest in writing backup images to disk for performance and availability reasons. The
use of EMC local and remote replication technologies including Replication Manager
simplify and enhance performance of the backup process by allowing most of the heavy
work of creating the actual operational backup to be offloaded from the production
database service.
Perhaps the most critical, but often overlooked, component of the backup process is
verification of the database once the backup process has completed. Important and often
difficult management tasks in the recovery process include:
♦ Ensuring that database image is complete and available in the event it is needed for
a recovery and has not lost any transactions.
♦ Integrating the database recovery process with application information, outside data
files, or other databases or applications.
Verifying the database after a backup depends on the customer’s specific applications,
requirements, and environment.
The paramount consideration for any backup and recovery processes is the need for
tested and well-documented backup and recovery procedures. Planning for,
documenting, and testing the required backup procedures for a particular database
environment is an essential part of maintaining a workable recovery strategy. In many
cases, tested and documented procedures for both backup and recovery are not available
in customer’s IT environments. Without these tried and documented procedures,
unforeseen issues can arise (such as the loss of key employees) or catastrophic failures
can occur, causing significant deleterious affects to critical business systems.

4.3.1 Online (hot) versus offline (cold) backups
The ability to create database images made consistent during recovery while both read
and write transactions are processing is a key differentiating feature of Oracle. Backup
images made while the database is hot are critical in environments with stringent uptime
requirements. By comparison, shutting down the database provides customers with a
safe and consistent method of creating backup database images at the expense of
database service outages. Choosing which of these user-managed backup methods to
implement depends on the customer’s needs and database environment.
Hot backups allow for greater availability, but also create more complex recovery
strategies as all logs containing active transactions must be present at recovery time to be
successful. Cold backups make for simple restore and recovery, but reduce the
availability of the system. Prolonged database service shutdowns to accommodate the
creation of extremely large database backups are frequently unacceptable to the
business. In these cases, online backups must be performed.
Making a hot copy of the database is now the standard, but this method has its own
challenges. How can a consistent copy of the database and supporting files be made
when they are changing throughout the duration of the backup? What exactly is the
content of the tape backup at completion? The reality is that the tape data is a “fuzzy

4-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

image” of the disk data, and considerable expertise is required to restore the database
back to a database point of consistency.
Online backups are made when the database is running in log archival mode. While
there are performance considerations for running in archive log mode, the overhead
associated with it is generally small compared with the enhanced capabilities and
increased data protection afforded by running in it. Except in cases such as large data
warehouses where backupsare unnecessary, or in other relatively obscure cases, archive
log mode is generally considered a best practice for all Oracle database environments.

4.3.2 Point-in-time and roll-forward recovery backups
Until recently, conditioning the database either through shutting down the database or
putting the tablespaces into hot backup mode was the only way to make Oracle database
images used for recovery. However, the requirement to recover databases and
applications in federated environments has driven the need to create new methods of
backing up databases and applications. Enabled through EMC consistencytechnologies,
point-in-time backup images, rather than fully recoverable copies have become
increasing important in customers’ complex federated environments. Consistently split
technologies in both local and remote EMC replication solutions provide a means of
creating dependent-write consistent database images made transactionally consistent
through the database’s own recovery mechanisms.
One backup method gaining increasing usage is combining EMC
consistencytechnologies with Oracle hot backup mode when creating backup images.
Using the two technologies together provides enhanced flexibility during the recovery
process since both restartable and recoverable databases are supported when this process
is used.
Currently, using the EMC consistency technology to create a recoverable database image without
conditioning Oracle is not supported.

4.3.3 Comparing partial and entire database backups
Backing up all the datafiles that compose the database is the typical method of doing
database backups. In some cases, however, backing up only pieces of the database, for
example the datafiles that make up a single tablespace make sense. One example of this
is the read-only tablespace. In some environments, a read-only tablespace is created
after all updates to it are complete. Monthly orders or sales transactions after month-end
processing are examples of where a tablespace might be converted from read/write to
read-only. Once a full backup of the tablespace is available, there is no need to continue
backups of that particular tablespace. In this case, taking a tablespace backup and saving
it once can save on subsequent database backup time, complexity, and costs. Although
in most cases full backups are customary, partial backups in certain situations are more
practical and effective.

4.3.4 Comparing incremental and full database backups
Making full database images for backup purposes is the standard method of backing up
databases. Creating incremental database backups is unavailable to users without the
use of RMAN. Additionally, incremental backups add complexity and create recovery

Oracle backup overview

4-7

Backing Up Oracle Environments

challenges when compared to full backups. Incremental backups use less space for the
backed up data, and in the latest releases of the database (Oracle10g Release 2), by
keeping a bitmap of changed tracks, incremental backups eliminate the need to fully
scan each database datafile.

4.4 Using EMC replication in the Oracle backup process
EMC disk-based replication is used to make a copy of an Oracle database and this copy
is used as a source for backup. A database backup process using disk replication
technology typically includes some or all of the following steps, depending on the
copying mechanism selected and the desired usage of the database backup:
♦ Preparing the array for replication
♦ Conditioning the source database
♦ Making a copy of the database volumes
♦ Resetting the source database
♦ Presenting the target database copy to a backup server
♦ Backing up the target database copy
In all cases but one, operating-system capabilities are used to back up the copies of the
database directories and containers. In other words, the Oracle backup utility RMAN is
not used except in the case described in Section 4.11.
The first step in the backup process depends on what method is going to be used to copy
the database volumes and whether it is required to “condition” the Oracle database in
any way. Conditioning could involve shutting down the database, or putting the
database in hot backup mode. How the backup is processed and subsequently restored
depends on what condition the database was in when the database copy was made. The
database can be in one of three data states when it is being copied:
♦ Shutdown
♦ Processing normally
♦ Conditioned using hot backup mode
Depending on the data state of the database at the time it is copied, the database copy
may be restartable or recoverable. While a restartable database is used in a valid
backup/restore strategy, it cannot guarantee some data loss. Most DBAs will want to
make a recoverable copy of the database such that logs taken after the backup areused to
roll forward the database to a point in time after the backup was taken. It is important to
understand that the database copies created with the EMC Symmetrix Storage array can
be used in a recoverable, roll-forward fashion, only if the database was conditioned
properly (hot backup mode or shut down) when the copy was created. In addition, the
way the restore is executed depends on the state of the database at the time the copy was
made. Chapter 5 covers the restore of the database.

4-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

The following sections describe how to make a copy of the database using three different
EMC technologies with the database in the three different states described in the prior
paragraph.
The primary method of creating copies of an Oracle database is through the use of the
EMC local replication product TimeFinder. TimeFinder is also used by Replication
Manager to make database copies. Replication Manager facilitates the automation and
management of database copies.
TimeFinder comes in three different forms: TimeFinder/Mirror, TimeFinder/Clone and
TimeFinder/Snap. These were discussed in general terms in section 2. Here, they are
used in a database backup context.

4.5 Copying the database with Oracle shutdown
Ideally, a copy of an Oracle database should be taken while the database is shut down.
Taking a copy after the database has shut down normally ensures a clean copy for
backups. In addition, a cold copy of a database is in a known transactional data state
which, for some application requirements, is exceedingly important. Copies taken of
running databases are in unknown transactional data states.
While a normal shutdown is desirable, it is not always feasible with an active Oracle
database. In many cases, applications and databases are forced to completely shut down.
Rarely, the shutdown abort command, which terminates all database engine processes
abruptly, may be required to successfully shut down the database. Whenever an
abnormal database shutdown occurs, it is recommended that the database be restarted
allowing the Oracle database engine to properly recover and clean up the database, and
then be shut down normally. This ensures a clean, consistent copy of the database is
available for the backup procedure.

4.5.1 Creating cold Oracle backup copies using TimeFinder/Mirror
TimeFinder/Mirror is an EMC software product that allows an additional storage
hardware mirror to be attached to a source volume. The additional mirror is a specially
designated volume in the Symmetrix configuration called a BCV. The BCV is
synchronized to the source volume through a process called an establish. While the
BCV is established, it is not accessible from any hosts. At an appropriate time, the BCV
can be split from the source volume to create a complete point-in-time copy of the
source data that can be used for backup.
Groups of standards and BCVs are managed together using SYMCLI device or
composite groups. Solutions Enabler commands are executed to create SYMCLI groups
for TimeFinder/Mirror operations. If the database spans more than one Symmetrix unit,
a composite group is used. Appendix B provides examples of these commands.
Figure 4-2 on page 4-10 shows how to use TimeFinder/Mirror to make a database copy
of a cold Oracle database.

Copying the database with Oracle shutdown

4-9

Backing Up Oracle Environments

Figure 4-2

Copying a cold Oracle database with TimeFinder/Mirror

1. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g device_group establish –full -noprompt

Note that the first iteration of the establish needs to be a full synchronization.
Subsequent iterations by default are incremental if the –full keyword is omitted.
When the command is issued, the array begins the synchronization process using
only Symmetrix resources. Since this operation occurs independently from the host,
the process must be interrogated to see when it completes. The command to
interrogate the synchronization process is:
symmir –g device_group verify

This command will return a 0 return code when the synchronization operation is
complete. Alternatively, synchronization is verified using the following:
symmir -g device_group query

After all of the volumes in the device group appears as synchronized, the split
command is issued at any time.
2. Once BCV synchronization is complete, bring down the database to make a copy of
a cold database:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

4-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

3. When the database is shut down, split the BCV mirrors using the following
command:
symmir –g device_group split -noprompt

The split command takes a few seconds to process. The database copy on the BCVs
is now ready for further processing.
4. The source database can now be activated and made available to users once again.
SQL> startup;

4.5.2 Creating cold Oracle backup copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume must be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix system, a
composite group is used. Appendix B provides examples of these commands.
Figure 4-3 shows how to use TimeFinder/Clone to make a copy of a cold Oracle
database onto the BCV devices.

Figure 4-3

Copying a cold Oracle database with TimeFinder/Clone

Copying the database with Oracle shutdown

4-11

Backing Up Oracle Environments

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g device_group create -noprompt

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior synchronization of data is necessary. After the
TimeFinder/Clone session is created it can be activated consistently.
2. Once the create command has completed, shut down the database to make a cold
disk copy of the database:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

3. With the database down, activate the TimeFinder /Clone:
symclone –g device_group activate -noprompt

After an activate command, the database copy provided by TimeFinder/Clone is
immediately available for further processing even though the copying of data may
not have completed.
4. Activate the source database to make it available to users again:
SQL> startup;

Databases copied using TimeFinder/Clone are subject to COFW and COA penalties.
The COFW penalty means that if a track is written to the source volume and has not
been copied to the target volume, it must first be copied to the target volume before the
write from the host is acknowledged. COA means that if a track on a TimeFinder/Clone
volume is accessed before it was copied, it must first be copied from the source volume
to the target volume. This causes additional disk read activity on the source volumes
and could be a source of disk contention on busy systems.

4.5.3 Creating cold Oracle backup copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving, pointerbased copies of disk volumes using VDEVs and SAVDEVs. The VDEVs contain
pointers either to the source data (when it is unchanged) or to the SAVDEVs (when the
data has been changed). VDEVs are not a full copy of the source data and rely on the
source devices. If the source device becomes unavailable, the virtual device will not be
available as well. In addition, if the SAVEDEV area gets full, it will invalidate the
TimeFinder/Snap session.
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix system, a
composite group is used. Appendix B provides examples of these commands .

4-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-4 shows how to use TimeFinder/Snap to make a copy of a cold Oracle
database.

Figure 4-4

Copying a cold Oracle database with TimeFinder/Snap

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g device_group create -noprompt

2. Once the create operation has completed, shut down the database in order to make a
cold TimeFinder/Snap of the DBMS. Execute the following Oracle commands:
sqlplus “/ as sysdba”
SQL> shutdown immediate;

3. With the database down, the TimeFinder/Snap copy can be activated:
symsnap –g device_group activate -noprompt

After the activate, the pointer-based database copy on the VDEVs is available for
further processing.
4. The source database can be activated again. Use the following Oracle command:
SQL> startup;

Databases copied using TimeFinder/Snap are subject to a COFW penalty while the snap
is activated. The COFW penalty means that if a track is written to the source or target

Copying the database with Oracle shutdown

4-13

Backing Up Oracle Environments

volumes and has not been copied to the snap save area, it must first be copied to the save
area before the write from the host is acknowledged.

4.6 Copying a running database using EMC consistency technology
Running database systems can be copied while the databases are servicing applications
and users. The database copying techniques use EMC consistency technology combined
with an appropriate data copy process like TimeFinder/Mirror or TimeFinder/Clone.
TimeFinder/CG allows for the running database copy to be created in an instant through
use of the –consistent key word on the split or activate commands. The image created
in this way is in a dependent-write consistent data state and can be used as a restartable
copy of the database.
Database management systems enforce a principle of dependent-write I/O. That is, no
dependent-write will be issued until the predecessor write it is dependent on has
completed. This type of programming discipline is used to coordinate database and log
updates within a database management system and allows those systems to be restartable
in event of a power failure. Dependent-write consistent data states are created when
database management systems are exposed to power failures. Using EMC consistency
technology options during the database replication process also creates a database copy
that has a dependent-write-consistent data state. Chapter 2 provides more information
on EMC consistency technology.
Oracle databases can be copied while running and processing transactions. The
following sections describe how to copy a running Oracle database using TimeFinder
technology.

4.6.1 Creating restartable Oracle backup copies using TimeFinder/Mirror
Figure 4-5 on page 4-15 shows how to use TimeFinder/Mirror to make a copy of a
running Oracle database.

4-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-5

Copying a running Oracle database with TimeFinder/Mirror

1. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g device_group establish –full -noprompt

Note that the first iteration of the establish must be a full synchronization.
Subsequent iterations are incremental and do not need the –full keyword. Once the
command is issued, the array begins the synchronization process using only
Symmetrix resources. Since this operation occurs independently from the host, the
process must be interrogated to see when it completes. The command to interrogate
the synchronization process is:
symmir –g device_group verify

This command will return a 0 return code when the synchronization operation is
complete. Alternatively, synchronization can be verified using the following:
symmir -g device_group query

2. Once the volumes are synchronized, issue the split command:
symmir –g device_group split –consistent -noprompt

The –consistent keyword tells the Symmetrix to use ECA to momentarily suspend
writes to the disks while the split is being processed. The effect of this is to create a
point-in-time copy of the database on the BCVs. It is similar to the image created
when there is a power outage that causes the server to crash. This image is a
restartable copy. The database copy on the BCVs is then available for further
processing.

Copying a running database using EMC consistency technology

4-15

Backing Up Oracle Environments

Since there was no specific coordination between the database state and the execution of
the consistent split, the copy is taken independent of the database activity. Using this
technique, EMC consistency technology is used to make point-in-time backups of
multiple systems atomically, resulting in a consistent point in time with respect to all
applications and databases included in the consistent split.

4.6.2 Creating restartable Oracle backup copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume needs to be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix system, a
composite group is used. Appendix B provides examples of these commands.
Figure 4-6 show how to use TimeFinder/Clone to make a copy of a running Oracle
database onto BCV devices.

Figure 4-6

Copying a running Oracle database using TimeFinder/Clone

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g device_group create -noprompt

4-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior copying of data is necessary.
2. After the TimeFinder/Clone relationship is created, it can be activated consistently:
symclone –g device_group activate –consistent -noprompt

The –consistent keyword tells the Symmetrix to use ECA to momentarily suspend
writes to the source disks while the TimeFinder/Clone is being activated. The effect
of this is to create a point-in-time copy of the database on the target volumes. It is a
copy similar in state to that created when there is a power outage resulting in a
server crash. This copy is a restartable copy. After the activate command, the
database copy on the TimeFinder/Clone devices is available for further processing.
Since there was no specific coordination between the database and the execution of the
consistent split, the copy is taken independent of the database activity. In this way,
EMC consistency technology is used to make point-in-time copies of multiple systems
atomically, resulting in a consistent point in time with respect to all applications and
databases included in the consistent split.
Databases copied using TimeFinder/Clone are subject to COFW and COA penalties.
The COFW penalty means that if a track is written to the source volume and has not
been copied to the target volume, it must first be copied to the target volume before the
write from the host is acknowledged. COA means that if a track on a target volume is
accessed before it is copied, it has to be copied from the source volume to the target
volume first. This causes additional disk read activity to the source volumes and could
be a source of disk contention on busy systems.

4.6.3 Creating restartable Oracle backup copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving pointer-based
copies of disk volumes using VDEVs and SAVDEVs. The VDEVs contain pointers
either to the source data (when it is unchanged) or to the SAVDEVs (when the data
changed).
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix unit, a
composite group is used. Appendix B provides examples of these commands.
Figure 4-7 on page 4-18 shows how to use TimeFinder/Snap to make a copy of a
running Oracle database.

Copying a running database using EMC consistency technology

4-17

Backing Up Oracle Environments

Figure 4-7

Copying a running Oracle database with TimeFinder/Snap

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g device_group create -noprompt

After the TimeFinder/Snap devices are created, all the pointers from the VDEVs
point at the source volumes. No data has been copied at this point. The snap can be
activated consistently using the consistent activate command.
2. Once the create operation has completed, execute the activate command with the
–consistent option to perform the consistent snap:
symsnap –g device_group activate –consistent -noprompt

The –consistent keyword tells the Symmetrix to use ECA (Enginuity Consistency
Assist) to momentarily suspend writes to the disks while the activate command is
being processed. The effect of this is to create a point-in-time copy of the database
on the VDEVs. It is similar to the state created when there is a power outage that
causes the server to crash. This image is a restartable copy. The database copy on
the VDEVs is available for further processing.
Since there was no specific action coordination between the database and the execution
of the consistent split, the copy is taken independent of the database activity. In this
way, EMC onsistency technology is used to make point-in-time copies of multiple
systems atomically, resulting in a consistent point in time with respect to all applications
and databases included in the consistent split.

4-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Databases copied using TimeFinder/Snap are subject to a COFW penalty while the snap
is activated. The COFW penalty means that if a track is written to the source volume
and has not been copied to the snap-save area, it must first be copied to the snap-save
area before the write from the host is acknowledged.

4.7 Copying the database with Oracle in hot backup mode
For many years, Oracle has supported hot backup mode, which provides the capability to
use split-mirroring technology while the database is online and create a recoverable
database on the copied devices. During this process, the database is fully available for
reads and writes. However, instead of writing change vectors (such as the rowid, before,
and after images of the data) to the online redo log, entire blocks of data are written.
These data blocks are then used to overwrite any potential inconsistencies in the data
files. While this enables the database to recover itself and create a consistent point-intime image after recovery, it also degrades performance while the database is in hot
backup mode.
An important consideration when using hot backup mode to create a copy of the
database is the need to split the archive logs separately from the database. This is
because Oracle must recover itself to the point after all of the tablespaces are taken out
of hot backup mode. If the hypervolumes containing the archive logs are split at the
same time as the data volumes, the marker indicating the tablespaces are out of hot
backup mode will not be found in the last archive log. As such, the archive logs must be
split after the database is taken out of hot backup mode, so the archive log devices (and
generally the redo logs as well) must be separate from the other data files.
The following sections describe the steps needed to put tablespaces or the entire
database into hot backup mode and take it out again. Appendix D provides a sample
script showing how hot backup mode can be used to create a recoverable Oracle
database image.

4.7.1 Putting the tablespaces or database into hot backup mode
To create a consistent image of Oracle while into hot backup mode, each of the
tablespaces in the database must be put into hot backup mode before copying can be
performed. The following command connects to the database instance and issues the
commands to put the tablespaces into hot backup mode:
sqlplus “/
SQL> alter
SQL> alter
SQL> alter
SQL> alter

as sysdba”
system archive log current;
tablespace DATA begin backup;
tablespace INDEXES begin backup;
tablespace SYSTEM begin backup;

Alternatively, with Oracle10g, the entire database can be put into hot backup mode with:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

Copying the database with Oracle in hot backup mode

4-19

Backing Up Oracle Environments

When these commands are issued, data blocks for the tablespaces are flushed to disk and
the datafile headers are updated with the last checkpoint SCN. Further updates of the
checkpoint SCN to the data file headers are not performed while in this mode. When
these files are copied, the nonupdated SCN in the datafile headers signifies to the
database that recovery is required.

4.7.2 Taking the tablespaces or database out of hot backup mode
To take the tablespaces out of hot backup mode, connect to the database and issue the
following commands:
sqlplus “/
SQL> alter
SQL> alter
SQL> alter
SQL> alter

as sysdba”
tablespace DATA end backup;
tablespace INDEXES end backup;
tablespace SYSTEM end backup;
system archive log current;

When these commands complete, the database is returned to its normal operating state.
Alternatively, with Oracle10g, the entire database can be put into hot backup mode with:
sqlplus “/ as sysdba”
SQL> alter database end backup;
SQL> alter system archive log current;

The log file switch command is used to ensure that the marker indicating that the
tablespaces are taken out of hot backup mode is found in an archive log. Switching the
log automatically ensures that this record is found in a written archive log.

4.7.3 Creating hot Oracle backup copies using TimeFinder/Mirror
TimeFinder/Mirror is an EMC software product that allows an additional hardware
mirror to be attached to a source volume. The additional mirror is a specially designated
volume in the Symmetrix configuration called a BCV. The BCV is synchronized to the
source volume through a process called an establish. While the BCV is established it is
not ready to all hosts. At an appropriate time, the BCV can be split from the source
volume to create a complete point-in-time copy of the source data that can be used for
multiple different purposes including backup, decision support, regression testing, etc.
Groups of BCVs are managed together using SYMCLI device or composite groups.
Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Mirror operations. If the database spans more than one Symmetrix system,
a composite group is used. Appendix B provides examples of these commands.
Figure 4-8 on page 4-21 shows how to use TimeFinder/Mirror to make a copy of an
Oracle database in hot backup mode.

4-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-8

Copying an Oracle database in hot backup mode with TimeFinder/Mirror

1. Establish the BCVs to the standard devices. This operation occurs in the
background and should be executed in advance of when the BCV copy is needed.
symmir –g data_group establish –full -noprompt
symmir –g log_group establish –full -noprompt

Note that the first iteration of the establish needs to be a “full” synchronization.
Subsequent iterations are incremental and do not need the –full keyword. Once the
command is issued, the array begins the synchronization process using only
Symmetrix resources. Since this is asynchronous to the host, the process must be
interrogated to see when it is finished. The command to interrogate the
synchronization process is:
symmir –g data_group verify
symmir –g log_group verify

This command will return a 0 return code when the synchronization operation is
complete.
2. When the volumes are synchronized, put the database in hot backup mode. Connect
to the database and issue the following commands:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute a split of the standard and BCV relationship:
symmir –g data_group split -noprompt

Copying the database with Oracle in hot backup mode

4-21

Backing Up Oracle Environments

The –consistent keyword is not used here as consistency is provided by the
database. The Data BCV(s) now contain an inconsistent copy of the database that
can be made consistent through recovery procedures using the archive logs. This is
a recoverable database. Usage of recoverable copies of databases is described in
Section 3.2.1.
4. After the replicating process completes, take the database (or tablespaces) out of hot
backup mode on the source database:
SQL> alter database end backup;
SQL> alter system archive log current;

5. After the tablespaces are taken out of hot backup mode and a log switch is
performed, split the Log BCV devices from their source volumes:
symmir –g log_group split -noprompt

4.7.4 Creating “hot” Oracle backup copies using TimeFinder/Clone
TimeFinder/Clone is an EMC software product that copies data internally in the
Symmetrix array. A TimeFinder/Clone session is created between a source data volume
and a target volume. The target volume needs to be equal to or greater in size than the
source volume. The source and target for TimeFinder/Clone sessions can be any
hypervolumes in the Symmetrix configuration.
TimeFinder/Clone devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Clone operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix B provides examples of these commands.
Figure 4-9 on page 4-23 shows how to use TimeFinder/Clone to make a copy of an
Oracle database in hot backup mode onto BCV devices.

4-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-9

Copying an Oracle database in hot backup mode with TimeFinder/Clone

1. Create the TimeFinder/Clone pairs. The following command creates the
TimeFinder/Clone pairings and protection bitmaps. No data is copied or moved at
this time:
symclone –g data_group create -noprompt
symclone –g log_group create -noprompt

Unlike TimeFinder/Mirror, the TimeFinder/Clone relationship is created and
activated when it is needed. No prior copying of data is necessary.
2. Place the Oracle database in hot backup mode:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute an “activate” of the TimeFinder/Clone:
symclone –g data_group activate -noprompt

The –consistent keyword is not used here as consistency is provided by the
database. The Data clone devices now contain an inconsistent copy of the database
that can be made consistent through recovery procedures using the archive logs.
This is a recoverable database. Section 3.7.2 describes use of recoverable copies of
databases.
4. After the replicating process completes, take the database (or tablespaces) out of hot
backup mode on the source database:

Copying the database with Oracle in hot backup mode

4-23

Backing Up Oracle Environments

SQL> alter database end backup;
SQL> alter system archive log current;

5. After the tablespaces are taken out of hot backup mode and a log switch is
performed, activate the Log clone devices:
symclone –g log_group activate -noprompt

Databases copied using TimeFinder/Clone are subject to COFW and COA penalties.
The COFW penalty means the first time a track is written to the source volume and has
not been copied to the target volume, it must first be copied to the target volume before
the write from the host is acknowledged. Subsequent writes to tracks already copied, do
not suffer from the penalty. COA means that if a track on a target volume is accessed
before it is copied, it must first be copied from the source volume to the target volume.
This causes additional disk read activity to the source volumes and could be a source of
disk contention on busy systems.

4.7.5 Creating “hot” Oracle backup copies using TimeFinder/Snap
TimeFinder/Snap enables users to create complete copies of their data while consuming
only a fraction of the disk space required by the original copy.
TimeFinder/Snap is an EMC software product that maintains space-saving pointer-based
copies of disk volumes using VDEVs and SAVDEVs. The VDEVs contain pointers
either to the source data (when it is unchanged) or to the SAVDEVs (when the data has
changed).
TimeFinder/Snap devices are managed together using SYMCLI device or composite
groups. Solutions Enabler commands are executed to create SYMCLI groups for
TimeFinder/Snap operations. If the database spans more than one Symmetrix, a
composite group is used. Appendix B provides examples of these commands.
The following diagram depicts the necessary steps to make a copy of an Oracle database
in hot backup mode using TimeFinder/Snap:

4-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-10

Copying an Oracle database in hot backup mode with TimeFinder/Snap

1. Create the TimeFinder/Snap pairs. The following command creates the
TimeFinder/Snap pairings and protection bitmaps. No data is copied or moved at
this time:
symsnap –g data_group create -noprompt
symsnap –g log_group create -noprompt

Unlike TimeFinder/Mirror, the snap relationship is created and activated when it is
needed. No prior copying of data is necessary. The create operation establishes the
relationship between the standard devices and the VDEVs and it also creates the
protection metadata.
2. After the snaps are created, place the Oracle database in hot backup mode:
sqlplus “/ as sysdba”
SQL> alter system archive log current;
SQL> alter database begin backup;

3. Execute an “activate” of the TimeFinder/Snap for the data devices:
symsnap –g data_group activate -noprompt

The –consistent keyword is not used here because consistency is provided by the
database. The VDEVs (and possibly SAVDEVs) contain a pointer-based copy of
the database while it is in hot backup mode. This is a recoverable database copy.
Section 3.7.2 describes use of recoverable copies of Oracle databases.

Copying the database with Oracle in hot backup mode

4-25

Backing Up Oracle Environments

4. Once the snap activate process completes, take the database (or tablespaces) out of
hot backup mode on the source database:
SQL> alter database end backup;
SQL> alter system archive log current;

5. After the database is taken out of hot backup mode and a log switch is performed,
activate the lsnap devices:
symsnap –g log_group activate -noprompt

Databases copied using TimeFinder/Snap are subject to a COFW penalty while the snap
is activated. The COFW penalty means that if a track is written to the source volume
and has not been copied to the snap-save area, it must first be copied to the save area
before the write from the host is acknowledged.

4.8 Backing up the database copy
The most common method of backing up an array-based copy of the database is to
present the volumes that contain the database copy to a backup server, import the
volume group, mount file systems, etc. When the backup server has access to the
volumes in this way, it can simply execute a file system or device backup of the database
volumes. Note that this backup is done at the OS level and uses OS utilities or a Tape
Management system utility to create the backup on tape.
Regardless of the tool used to create the backup copy and regardless of the state of the database at
the time the copy was created, the backup process is the same, except as noted in the next section.

Appendix C describes issues around accessing, importing, and mounting copies of
database volumes.

4.9 Backups using EMC Replication Manager for Oracle backups
EMC Replication Manager (RM) is used to manage and control the TimeFinder copies
of an Oracle database used for backups. It also may be integrated with backup products
such as NetWorker. The RM product has a GUI and command line and provides the
capability to:
♦ Autodiscoverthe standard volumes holding the database
♦ Identify the pathname for all database files
♦ Identify the location of the archive log directories
♦ Identify the location of the database binaries, dump files, etc.
♦ Callouts to integrate backup utilities with the database copies
Using this information, RM can set up TimeFinder Groups with BCVs or VDEVs,
schedule TimeFinder operations and manage the creation of database copies, expiring
older versions as needed.

4-26

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

Figure 4-11 demonstrates the steps performed by RM using TimeFinder/Mirror to create
a database copy to use for multiple other purposes:

Figure 4-11

Using RM to make a TimeFinder copy of Oracle

RM does the following:
1. Logs in to the database and discovers the locations of all the datafiles and logs on
the Symmetrix devices. Note that the dynamic nature of this activity will handle the
situation when extra volumes are added to the database. The procedure will not
have to change.
2. Establishes the standards to the BCVs in the Symmetrix system. RM polls the
progress of the establish process until the BCVs are synchronized, and then moves
on to the next step.
3. Performs a log switch to flush changes to disk, minimizing recovery required of the
copied database.
4. Puts the Oracle database in hot backup mode, discussed in Section 4.7.1.
5. Issues a TimeFinder split, to detach the data BCVs from the standard devices.
6. Takes the Oracle database out of hot backup mode, as discussed in Section 4.7.2.
7. Performs another log switch to flush the end of hot backup mode marker from the
online redo logs to an archive log.
8. Creates a copy of a backup control file.
9. Copies the backup controlfile and additional catalog information the RM host.

Backups using EMC Replication Manager for Oracle backups

4-27

Backing Up Oracle Environments

10. Copies the database archive logs to the replication manager host for use in the
restore process.

4.10 Backups using Oracle Recovery Manager (RMAN)
Oracle Recovery Manager, or RMAN, is an Oracle backup and recovery utility first
available in Oracle 8. RMAN contains a command-line client and integration into the
Enterprise Manager (OEM) GUI. The utility integrates with procedures and sessions
running on the Oracle servers in order to manage the backup and recovery processes. In
addition, RMAN can create a backup repository that contains information about all
backups and recoveries in the environment.
RMAN provides a number of key benefits over traditional user managed database
backups. They include:
♦ Automated backup and recovery – The RMAN utility provides functionality that
can automate backup and recovery processes to tape or disk.
♦ Backup catalogs – RMAN monitors all database backup activities and stores
information concerning backed up datafiles and archive logs in a centrally managed
repository.
♦ Incremental backup support – RMAN enables the ability to create incremental
backup images that shorten the backup window and reduce the amount of tape or
disk space needed for backups.
♦ Block level corruption detection and recovery – During the backup process,
RMAN reads each of the data blocks and determines whether the block is
consistent. If data corruption issues exist, RMAN can provide block media
recovery to correct any corruption issues.
♦ Hot backup mode not required for hot backups – RMAN provides the ability to
create a database backup image that can be made consistent without having to put
the database into hot backup mode (assumes that primary database is used for
backups rather than a database copy).
♦ Integration with OEM – Oracle Enterprise Manager provides a GUI interface that
simplifies the management process of Oracle databases. RMAN integrates with
OEM to provide a single location for managing all aspects of multiple database
environments.
♦ Backup types – RMAN provides the ability to back up databases to either tape or
disk. It also integrates with specialized media managers that assist in simplifying
the backups of Oracle with backup solutions including NetWorker and VERITAS
NetBackup.
RMAN provides a broad range of backup options and features. Describing these
features in detail is beyond the scope of this document. For additional detailed
information on RMAN, consult the Oracle documentation Oracle Database Backup and
Recovery Basics and Oracle Database Backup and Recovery Advanced User’s Guide.

4-28

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Backing Up Oracle Environments

4.11 Backups using TimeFinder and Oracle RMAN
It is possible to back up an Oracle database image made with TimeFinder using the
RMAN backup utility. This option retains all of the benefits of RMAN while
simultaneously offloading the resources needed by Oracle’s backup utility to process the
production database blocks. In addition, using TimeFinder in conjunction with RMAN
enables recovery procedures to be tested and validated before affecting production data,
since a second copy of the data is available.

Backups using TimeFinder and Oracle RMAN

4-29

Backing Up Oracle Environments

4-30

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 5

Restoring and
Recovering Oracle
Databases

This chapter presents these topics:
5.1
5.2
5.3
5.4
5.5
5.6
5.7

Oracle recovery types ......................................................................................... 5-2
Oracle recovery overview................................................................................... 5-5
Restoring a backup image using TimeFinder ..................................................... 5-6
Restoring a backup image using Replication Manager .................................... 5-14
Oracle database recovery procedures ............................................................... 5-15
Database recovery using Oracle RMAN .......................................................... 5-20
Oracle Flashback .............................................................................................. 5-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

5-1

Restoring and Recovering Oracle Databases

Recovery of a production database is an event that all DBAs hope is never required.
Nevertheless, DBAs must be prepared for unforeseen events such as media failures or
user errors requiring database recovery operations. The keys to a successful database
recovery include the following:
♦ Identifying database recovery time objectives
♦ Planning the appropriate recovery strategy based upon the backup type (full,
incremental)
♦ Documenting the recovery procedures
♦ Validating the recovery process
Oracle recovery depends on the backup methodology used. With the appropriate backup
procedures in place, an Oracle database is recovered to any point in time between the
end of the backup and the point of failure using a combination of backed up data files
and Oracle recovery structures including the control files, the archive logs, and the redo
logs. Recovery typically involves copying the previously backed up files into their
appropriate locations and, if necessary, performing recovery operations to ensure that the
database is recovered to the appropriate point in time and is consistent.
The following sections examine both traditional (user-managed) and RMAN Oracle
database recoveries. This chapter assumes that EMC technology is used in the backup
process as described in Chapter 4. Thus, this chapter directly matches the sections of
that chapter.

5.1 Oracle recovery types
There are several reasons for Oracle to perform recovery. Examples include recovery of
the database after a power failure on the host, recovery after user error that deletes a
required part of the database, recovery after application errors, and recovery on account
of hardware (disk, host, HBA, and such) or software failures or corruptions.
The following sections discuss the various types of recovery operations available in
Oracle, under what circumstances they are employed, and high-level details of each
recovery operation. These operations are then further discussed in subsequent sections
of this chapter.

5.1.1 Crash recovery
A critical component of all ACID-compliant (Atomicity Consistency Isolation
Durability) databases is the ability to perform crash recovery to a consistent database
state after a failure. Power failures to the host are a primary concern for databases to go
down inadvertently and require crash recovery. Other situations where crash recovery
procedures are needed include databases shut down with the “abort” option and database
images created using a consistent split mechanism.
Crash recovery is an example of using the database restart process, where the implicit
application of database logs during normal initialization occurs. Crash recovery is a
database driven recovery mechanism—it is not initiated by a DBA. Whenever the

5-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

database is started, Oracle verifies that the database is in a consistent state. It does this
by reading information out of the control file and verifying the database was previously
shut down cleanly. It also determines the latest checkpoint system change number
(SCN) in the control file and verifies that each datafile is current by comparing the SCN
in each data file header. In the event that a crash occurred and recovery is required, the
database automatically determines which log information needs to be applied. The latest
redo log is read and change information from them is applied to the database files,
rolling forward any transactions that were committed but not applied to the database
files. Then, any transaction information written to the datafiles, but not committed, are
rolled back using data in the undo logs.
In addition to enabling Oracle databases to recover after an outage, crash recovery is
also essential to restartable databases that use the EMC consistency technologies. These
consistency technologies in conjunction with TimeFinder, SRDF, or Open Replicator,
enable dependent-write consistent database images to be created. When these copies are
restarted, the database uses crash recovery mechanisms to transform the dependent-write
consistent images into transactionally consistent database images.

5.1.2 Media recovery
Media recovery is another type of Oracle recovery mechanism. Unlike crash recovery
however, media recovery is always user-invoked, although both user-managed and
RMAN recovery types may be used. In addition, media recovery rolls forward changes
made to the datafiles restored from disk or tape due to their loss or corruption. Unlike
crash recovery, which uses only the online redo log files, media recovery uses both the
online redo logs and the archived log files during the recovery process.
The granularity of a media recovery depends on the requirements of the DBA. It can be
performed for an entire database, for a single tablespace, or even for a singledatafile.
The process involves restoring a copy of a valid backed up image of the required data
structure (database, tablespace,datafile) and using Oracle standard recovery methods to
roll forward the database to the point in time of the failure by applying change
information found in the archived and online redo log files. Oracle uses SCNs to
determine the last changes applied to the data files involved. It then uses information in
the control files that specifies which SCNs are contained in each of the archive logs to
determine where to start the recovery process. Changes are then applied to appropriate
datafiles to roll them forward to the point of the last transaction in the logs.
Media recovery is the predominant Oracle recovery mechanism. Media recovery is also
used as a part of replicating Oracle databases for business continuity or disaster recovery
purposes. Further details of the media recovery process are in the following sections.

5.1.3 Complete recovery
Complete recovery is the primary method of recovering an Oracle database. It is the
process of recovering a database to the latest point in time (just before the database
failure) without the loss of committed transactions. The complete recovery process
involves restoring a part or all of the database data files from a backup image on tape or
disk, and then reading and applying all transactions subsequent to the completion of the
database backup from the archived and online log files. After restarting the database,

Oracle recovery types

5-3

Restoring and Recovering Oracle Databases

crash recovery is performed to make the database transactionally consistent for
continued user transactional processing.
The processes needed to perform complete recovery of the database are detailed in the
following sections.

5.1.4 Incomplete recovery
Oracle sometimes refers to incomplete database recovery as a point-in-time recovery.
Incomplete recovery is similar to complete recovery in the process used to bring the
database back to a transactionally consistent state. However, instead of rolling the
database forward to the last available transaction, roll-forward procedures are halted at a
user-defined prior point. This is typically done to recover a database prior to a point of
user error such as the deletion of a table, undesired deletion or modification of customer
data, or rollback of an unfinished batch update. In addition, incomplete recovery is also
performed when recovery is required, but there are missing or corrupted archive logs.
Incomplete recovery always incurs some data loss.
Typically, incomplete recovery operations are performed on the entire database since
Oracle needs all database files to be consistent with one another. However, an option
called Tablespace Point-in-Time Recovery (usually abbreviated TSPITR), which allows
a single tablespace to be only partially recovered, is also available. This recovery
method, in Oracle10g, uses the transportable tablespace feature described in Section 3.8.
The Oracle documentation Oracle Database Backup and Recovery Advanced Users
Guide provides additional information on TSPITR.
The need to perform incomplete recovery to fix user errors has become less important
with the introduction of the Oracle Flashback capabilities. Flashback functionality is
described in Section 5.7.

5.1.5 Restartable database recovery
In addition to the more commonly used complete and incomplete recovery methods,
another database recovery scheme gaining increasing relevance is restartable database
recovery. Restartable database recovery is differentiated from incomplete recovery by
the fact that the EMC consistency technology is required to make a restartable image,
rather than one using Oracle’s recovery procedures.
Using this method, the entire database image written to tape or disk is restored to the
point of the backup and the database service is then restarted. At restart, Oracle
performs crash recovery, rolling forward any changes that did not make it to the data
files and rolling back changes that had not committed. The database is then opened for
user activities.
While a restartable database recovery invariably involves the loss of data since the
backup was made at someprevious time, there are benefits to restoring restartable
database images. Primary among them is the ability to coordinate recovery points with
multiple, or federated database environments. The operational complexity of managing
the recovery points of multiple databases, related application data, or infrastructure
messaging queues can be difficult, if not impossible using traditional backup techniques.
Restart procedures in conjunction with EMC consistency technology allows customers

5-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

to create a point-in-time image of all applications and databases. This reduces or
eliminates operational complexity and enables reduced recovery times for complex
enterprise environments.

5.2 Oracle recovery overview
Oracle has two primary recovery methods: user-managed and RMAN. A traditional
user-managed recovery involves restoring data files from tape or disk using operating
system commands and performing Oracle recovery operations to create a consistent
database image to a specified point in time between the last available backup and the
point of failure. User-managed recoveries require careful consideration by the DBA to
ensure that the appropriate files (archive logs, control files, datafiles, and such) are
available and restored to their correct locations, and that the recovery process is
performed successfully.
An alternative to user-managed recovery is RMAN. After using RMAN for the backup
process, the utility also may be used to recover the database. RMAN maintains an
internal list of backup filenames and their locations in order to automate the recovery
process. RMAN is an automated, efficient utility that simplifies the Oracle recovery
process.
The following sections describe considerations for the various user-managed recovery
processes. These include:
♦ Online versus offline recovery – In most cases, recovery is performed by shutting
down the primary database, restoring one or more data files from tape or disk, and
recovering the datafile(s). This requires downtime for the database. An alternative
to this is recovering a single tablespace, which in some cases can be done online.
♦ Point-in-time versus roll-forward recovery – Historically, recovery procedures
involved restoring a copy of the database and using Oracle recovery mechanisms in
conjunction with the archive logs (and the last redo logs) to recover the database to
the point of a failure. In federated environments, however, restoring a database
image to a known state and using Oracle crash recovery procedures to make the
database consistent is a growing alternative to traditional roll-forward recovery.
♦ Partial (tablespace or datafile) versus full database recovery – Oracle provides the
ability to recover a single data file or tablespace in addition to recovering the entire
database. This option is useful for example if a single datafile becomes corrupted
or data from a single tablespace is lost.
♦ Incomplete versus complete database recovery – In general, customers want to
recover a database fully to the point of a failure. In some cases, however, due to
lost or corrupted archive logs from tape, incomplete recovery may necessary or
even desired.
The recovery process consists of three primary steps:
♦ Restoring a database backup (that is, the backed up datafiles or raw devices, from
either tape or disk)

Oracle recovery overview

5-5

Restoring and Recovering Oracle Databases

♦ Recovering the database using Oracle database recovery methods
♦ Verifying the state of the recovered database including consistency of the database,
recovery point, and coordination with other databases or applications
Most recovery procedures require both a restore of a database image and user-initiated
recovery of that image to a consistent state. However, this is not always the case. In
some circumstances, simply restoring an image of the database and restarting the
instance is all that is needed to bring the database to a consistent, defined state and
continue operations. Planning for, documenting, and testing the required recovery
procedures for a particular database environment are an essential part of maintaining a
workable recovery strategy.
Perhaps the most critical, but often overlooked, component of the recovery process is
verification of the database once the restore and recovery stepsare complete. Important
and often difficult management tasks in the recovery process include:
♦ Ensuring that the database is consistent and has not lost any transactions.
♦ Integrating the database recovery process with application information, datafiles
stored outside of the database, or other databases or applications.
Verifying the database after restore and recovery depends upon the customer’s specific
applications, requirements, and environment. As such, it is not discussed further in this
document.

5.3 Restoring a backup image using TimeFinder
The first step in any recovery process is to restore a backup image from either tape or
disk media. In this section, the copy is on disk media and was created by TimeFinder. It
is assumed that the disks contain the database image needed for restore. The exact
restore process depends on how the copy on disk was created. If the database image
needed for the restore is on tape, the procedures are different and beyond the scope of
this document. Ideally, a copy of an Oracle database shut down during the backup
process is available for restore, as it provides the greatest flexibility for the recovery
processing. However, backup images taken while the database was in hot backup mode
or was not conditioned in any way all use similar restore procedures.
Restore operations that use only Symmetrix array resources may be performed if
EMC TimeFinder was used to create a backup image of the database. If a database copy
was made while the database was shut down, this copy can be restored (with
TimeFinder, either incrementally or fully) and used as a point-in-time image, or as part
of an incomplete or complete recovery of the database. Alternatively, if the backup
image was made while the database was in a hot backup state, TimeFinder may also be
used to restore an inconsistent image of the database that can be successfully recovered
using Oracle incomplete or complete recovery techniques to a user defined point of
consistency.
TimeFinder comes in three different forms: TimeFinder/Mirror, TimeFinder/Clone and
TimeFinder/Snap. These were discussed in general terms in Chapter 2. Here, they are

5-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

used in a database recovery context. The following sections describe the restore process
when each variant of TimeFinder is used.

5.3.1 Restore using TimeFinder/Mirror
If TimeFinder/Mirror was used to create a backup database image, the TimeFinder
restore process can be used to copy the backup image back to the production volumes.
Two cases for the restore process exist. In the first case, when a point-in-time restartable
database image restore is desired, all of the data files that make up the database
including the archive logs, redo logs, and control files are restored from the BCV
devices. This first case is depicted in Figure 5-1 on page5-7, where both the volumes
containing the datafiles and the database recovery structures (archive logs, redo logs, and
control files) are restored.
Prior to any disk-based restore using EMC technology, the database must be shut down,
and file systems unmounted. The operating system should have nothing in its memory
that reflects the content of the database file structures.

Figure 5-1

Restoring a TimeFinder copy, all components

In most circumstances, only the datafiles (or even a subset of thedatafiles) are restored.
In these instances, user-initiated complete or incomplete database recoveries are
planned. Figure 5-2 on page 5-8 depicts the case where only the datafiles(or a subset of
thedatafiles) are restored.

Restoring a backup image using TimeFinder

5-7

Restoring and Recovering Oracle Databases

Figure 5-2

Restoring a TimeFinder copy, data components only

In the example that follows, the data_group device group holds all Symmetrix volumes
containing Oracle tablespaces. The log_group group has volumes containing the Oracle
recovery structures (the archive logs, redo logs, and control files). The following steps
describe the process needed to restore the database image from the BCVs:
1. Verify the state of the BCVs. All volumes in the Symmetrix device group should be
in a split state. The following commands identify the state of the BCVs for each of
the device groups:
symmir –g data_group query
symmir –g log_group query

2. Shut down the database on the production volumes. From a storage perspective, the
restore process does not require the database to shut down. However, because data
blocks are changing at the storage layer, Oracle is not aware of changes occuring
during the restore process. As such, data in the SGA may not be consistent with
data on disk. This inconsistency requires a brief outage of the database while the
restore process is initiated.
sqlplus “/ as sysdba”
SQL> shutdown immediate
SQL> startup restrict;
SQL> shutdown

3. After the primary database has shut down, unmount the file system (if used) to
ensure that nothing remains in cache. This action is operating-system dependent.
4. Once the primary database has shut down successfully and the file system is unmounted, initiate the BCV restore process. In this example, both the data_group

5-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

and log_group device groups are restored, indicating a point-in-time recovery. If an
incomplete or complete recovery is required, only the data_group device group
would be restored. Execute the following TimeFinder/Mirror SYMCLI commands:
symmir –g data_group restore –nop
symmir –g log_group restore –nop
symmir –g data_group query
symmir –g log_group query

5. Once the BCV restore process has been initiated, the production database copy is
ready for recovery operations. It is possible to start the recovery process even
though the data is still being restored from the BCV to the production devices. Any
tracks needed, but not restored, will be pulled directly from the BCV device. It is
recommended however, that the restore operation completes and the BCVs are split
from the standard devices before the source database is started and recovery (if
required) is initiated.
It is important to understand that if the database is restarted before the restore process
completes, any changes to the source database volumes will also be written to the BCVs.
This means that the copy on the BCV will no longer be a consistent database image. It is
always recommended that the restore process completes and the BCVs are split from the
source volumes before processing or recovery is initiated on the source devices.

6. After the restore process completes, split the BCVs from the standard devices with
the following commands:
symmir –g data_group split –nop
symmir –g log_group split –nop
symmir –g data_group query
symmir –g log_group query

5.3.2 Restore using TimeFinder/Clone
TimeFinder/Clone allows a target clone image to be restored back to the source device
or to an unrelated target device. Prior to Solutions Enabler 6.0, data could be restored
from a clone target back to its source device only by performing a reverse clone
operation. The clone relationship was terminated between the source and the target; the
target was then used as the source for creating and activating a new clone relationship.
However, with SYMCLI 6.0 running with Enginuity 5671 code, an operation similar to a
TimeFinder/Mirror restore can be performed without terminating the original clone
session. A prerequisite of this is that the clone operation must have been created with
the –differential option.
If TimeFinder/Clone is used to create a database backup image, the TimeFinder/Clone
restore process can be used to copy the backup image back to the production volumes.
Figure 5-3 on page 5-10 depicts the necessary steps to restore a database copy of an
Oracle database using TimeFinder/Clone. In this example, both the volumes containing
data files and database recovery structures (archive logs, redo logs, and control files) are
restored in anticipation that a point-in-time recovery (rather than a complete or
incomplete recovery) will be performed. Alternatively, if complete recovery is required,

Restoring a backup image using TimeFinder

5-9

Restoring and Recovering Oracle Databases

only the volumes containing the database datafiles may be restored, as is shown in
Figure 5-4 on page 5-10.

5-10

Figure 5-3

Restoring a TimeFinder/Clone copy, all components

Figure 5-4

Restoring a TimeFinder/Clone copy, data components only

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

In the example that follows, the data_group device group holds all Symmetrix volumes
containing Oracle tablespaces. The log_group group has volumes containing the Oracle
recovery structures (the archive logs, redo logs, and control files). Follow these steps to
restore the database image from the BCV clone devices:
1. Verify the state of the clone devices. Volumes in the Symmetrix device group
should be in an active state, although the relationship between the source and target
volumes may have terminated. The following commands identify the state of the
clones for each of the device groups (the –multi flag is used to show all
relationships available):
symclone –g data_group query –multi
symclone –g log_group query -multi

2. Shut down the database on the production volumes. From a storage perspective, the
restore process does not require the database to be shut down. However, because
data blocks are changing at the storage layer, Oracle is not aware of changes taking
place during the restore process. As such, data in the SGA isinconsistent with data
on disk. This inconsistency requires a brief outage of the database while the restore
process is initiated.
sqlplus “/ as sysdba”
SQL> shutdown immediate
SQL> startup restrict;
SQL> shutdown

3. After the primary database has shut down, unmount the file system (if used) to
ensure that nothing remains in server cache. This action is operating-system
dependent.
4. Initiate the clone restore process. In this example, both the data_group and
log_group device groups are restored, indicating a point-in-time recovery. If an
incomplete or complete recovery is required, only the data_group device group
would be restored. Execute the following TimeFinder/Clone SYMCLI commands:
symclone –g data_group restore –nop
symclone –g log_group restore –nop
symclone –g data_group query –multi
symclone –g log_group query –multi

5. After the clone restore process is initiated, the production database copy is ready for
recovery operations. It is possible to start the recovery process even though the data
is still being restored from the BCV to the production devices. Any tracks needed,
but not restored, are pulled directly from the BCV device.
6. After the restore process completes, terminate the clone/standard relationships as
follows:
symclone –g data_group terminate –nop
symclone –g log_group terminate –nop
symclone –g data_group query –multi
symclone –g log_group query –multi

Restoring a backup image using TimeFinder

5-11

Restoring and Recovering Oracle Databases

5.3.3 Restore using TimeFinder/Snap
TimeFinder/Snap allows a target virtual database image to be restored back to the source
device or to an unrelated target device. Prior to Solutions Enabler 5.4, when data was
restored from a snap back to its source device, any other snap sessions created were
terminated. Beginning with SYMCLI 5.4, restore operations using TimeFinder/Snap
can maintain the relationship between be the source device and any other snaps.
Additional snap sessions are persistent through a restore operation to the source device.
If TimeFinder/Snap were used to create a backup database image, the TimeFinder
restore process can be used to copy the backup image to the production volumes. Figure
5-5 on page 5-12 shows how to use TimeFinder/Clone to make a database copy of a cold
Oracle database. In this example, both the volumes containing data files and database
recovery structures (archive logs, redo logs, and control files) are restored in anticipation
that a point-in-time recovery (rather than a complete or incomplete recovery) will be
performed. Alternatively, if complete recovery is required, only the volumes containing
the database data files may be restored, as shown in Figure 5-6 on page 5-13.

Figure 5-5

5-12

Restoring a TimeFinder/Snap copy, all components

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

Figure 5-6

Restoring a TimeFinder/Snap copy, data components only

1. Verify the state of the snap devices. Volumes in the Symmetrix device group should
be in an active state, although the relationship between the source volumes and
virtual devices may also have been terminated. The following commands identify
the state of the clones for each of the device groups (the –multi flag is used to show
all relationships available):
symsnap –g data_group query –multi
symsnap –g log_group query -multi

2. Shut down the database on the production volumes. From a storage perspective, the
restore process does not require the database to be shut down. However, because
data blocks are changing at the storage layer, Oracle is unaware of changes occuring
during the restore process. As such, data in the SGA is inconsistent with data on
disk. This inconsistency requires a brief outage of the database while the restore
process is initiated.
sqlplus “/ as sysdba”
SQL> shutdown immediate
SQL> startup restrict;
SQL> shutdown

3. After the primary database shuts down, unmount the file system (if used) to ensure
that nothing remains in cache. This action is operating-system dependent.
4. Once the file systemsare unmounted, initiate the snap restore process. In this
example, both the data_group and log_group device groups are restored, indicating
a point-in-time recovery. If an incomplete or complete recovery is required, only the
data_group device group would be restored. Execute the following
TimeFinder/Clone SYMCLI commands:

Restoring a backup image using TimeFinder

5-13

Restoring and Recovering Oracle Databases

symsnap –g data_group restore –nop
symsnap –g log_group restore –nop
symsnap
symsnap
symsnap
symsnap

–g
–g
–g
–g

data_group query –restore
data_group query –multi
log_group query –restore
log_group query -multi

5. After the snap restore process is initiated, the production database copy is ready for
recovery operations. It is possible to start the recovery process even though the data
is still being restored from the BCV to the production devices. Any tracks needed,
but not restored, are pulled directly from the BCV device.
6. When the snap restore process is initiated, both the snap device and the source are
set to a Not Ready status (that is, they are offline to host activity). Once the restore
operation commences, the source device is set to a Ready state. Upon completion of
the restore process, terminate the restore operations as follows:
symsnap –g data_group terminate –restored -noprompt
symsnap –g log_group terminate –restored -noprompt
symsnap –g data_group query
symsnap –g log_group query
Note: Terminating the restore session does not terminate the underlying snap session.

7. If the snap device is needed for further processing once the restore process
completes, the virtual devices must be set to a Ready state again. This is
accomplished through the command:
symld –g data_group ready VDEV001 –vdev

and so on for each VDEV in the device group.
Alternatively, after the restore process completes, the snap/standard relationships
can be terminated with the following commands:
symsnap –g data_group terminate –noprompt
symsnap –g log_group terminate –noprompt
symsnap –g data_group query –multi
symsnap –g log_group query –multi

5.4 Restoring a backup image using Replication Manager
Replication Manager provides automated restore procedures through a graphical user
interface that simplifies the restore process. If Replication Manager is used to create a
replica of the database, a restore process can simply be initiated through the Replication
Manager interface.
Figure 5-7 on page 5-15 demonstrates the steps performed by Replication Manager
using TimeFinder/Mirror to restore a database copy so that recovery can be performed.
Note that Replication Manager has the ability to leave the database in a restored state so

5-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

the DBA can initiate recovery procedures or it can start the recovery process
automatically. Replication Manager has several options for restoring and recovering an
Oracle database.

Figure 5-7

Restoring Oracle using EMC Replication Manager

1. Shut down the primary Oracle database so that data can be restored.
2. Replication Manager initiates a restore operation to copy the backed up version of
the database back over the primary Oracle database’s datafiles.
3. A copy of the backup control file is copied from the Replication Manager host to the
Oracle host for recovery procedures.
4. Copies of the required archive logs are copied from the Replication Manager host to
the Oracle host for recovery procedures.
5. At this point, the database is ready for recovery. Depending on how it is configured,
recovery operations may be manually or automatically initiated.
For more information on Replication Manager capabilities with Oracle, consult the latest
Replication Manager Product Guide or Replication Manager Administrator’s Guide.

5.5 Oracle database recovery procedures
After the database is restored from either tape or disk, the next step in the process is to
perform recovery procedures to bring the database into a consistent state and open the
database for user transactions. The type of recovery implemented depends on the
backup process originally used. The recovery procedures required for each of the
restored database images discussed in Sections 5.3 and 5.4 are described next.

Oracle database recovery procedures

5-15

Restoring and Recovering Oracle Databases

5.5.1 Oracle restartable database recovery procedures
An Oracle point-in-time recovery here refers to the process needed to recover a backup
database image taken using one of the EMC consistency technologies. Recovery
examples for cold, hot backup mode, quiesced, and consistently split database backup
images are described next. For each case, the recovery process is managed by the
database itself through Oracle crash recovery procedures, rather than media recovery
operations.
5.5.1.1 Restartable database recovery from a cold backup image

Creating a point-in-time recovered database image usually requires Oracle crash
recovery procedures. However, after taking a cold backup image, no database recovery
is required to bring the datafiles into a consistent state. Ensure the restored database
files are available. The database can then be restarted using the commands:
sqlplus “/ as sysdba”
SQL> startup;
5.5.1.2 Restartable database recovery from a hot backup image

For customers with the ability to perform consistent TimeFinder splits, utilizing EMC
consistent splits with hot backup mode provides an effective way to create both
restartable and recoverable database images. Using both together maximizes the
flexibility of options should either restart or recovery be required.
A point-in-time recovery of a restored image taken with the source in hot backup mode
in general requires user-initiated recovery. To recover the hot backup image, follow
these steps:
1. Recover the database to the point when the tablespaces were taken out of hot backup
mode:
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

The recover database command initiates recovery procedures within Oracle. The
control files and each of the datafiles are examined. The database then lists the
appropriate archive logs to apply. Each archive log is then applied in turn to the
point when the tablespaces were taken out of hot backup mode.
2. Once recovery is complete, open the database to users:
SQL> alter database open;
Note: It is also possible to simply restart the database as shown in the next section.

5-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

5.5.1.3 Restartable database recovery from a consistent split backup image

Creating a point-in-time restartable database image only requires Oracle crash recovery
procedures to open it to users. Ensure the restored database files are available. The
database can then be restarted using the commands:
sqlplus “/ as sysdba”
SQL> startup;

5.5.2 Oracle complete recovery
A complete recovery requires that all committed transactions from the database are
applied to the restored database image. Oracle media recovery procedures are initiated
to determine whether any datafiles in the backup image need recovery. Oracle also
determines that archive logs to apply to roll forward the database to the state before the
failure occurred.
In most cases, the source database must be shut down for recovery procedures to be
initiated. In some cases however, for example when only a single datafile or tablespace
needs to be recovered, the process can be completed with the database open (only the
tablespace needs to be taken offline).
The following subsections describe the recovery procedures for Oracle complete
recovery. The type of recovery implemented depends on the backup process originally
used. The recovery procedures required for each of the restored database images
discussed in Sections 5.3 and 5.4 are described next.
5.5.2.1 Oracle complete recovery from a cold backup image

Complete recovery using a cold backup image of the database is easily managed as the
database is already in a consistent transactional state. SCN information in the control
file defines the recovery point for the database. Each of the datafiles in the database also
contains the latest SCN checkpoint. This information is compared and the set of archive
logs needed to roll forward the database is determined.
To recover the database from a cold backup image, follow these steps:
1. Recover the database to the point of the latest transactions.
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

The recover database command initiates recovery procedures within Oracle. The
control files and each of the datafiles are examined. The database then lists the
appropriate archive logs to apply. Each archive log is then applied in turn to the
point when the tablespaces were taken out of hot backup mode. Additionally, the
latest redo log information can be applied to the database by specifying the latest
logs.
2. Once recovery is complete, open the database to users:
SQL> alter database open;

Oracle database recovery procedures

5-17

Restoring and Recovering Oracle Databases

5.5.2.2 Oracle complete recovery for a hot backup image

In general, a point-in-time recovery of a restored image taken with the source in hot
backup mode requires user-initiated recovery. The following demonstrates the process
of recovering the hot backup image:
1. The database must be fully recovered to the point of database failure. This requires
that all archive logs and the latest redo logs are available.
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

The recover database command initiates recovery procedures within Oracle. The
control files and each of the datafiles are examined. The database then lists the
appropriate archive logs to apply. Information in each archive log is then applied to
the database in turn. This is a manual process, although it can be automated if all the
logs are in an appropriate directory (such as the flash recovery area) by specifying
auto when Oracle requests a specific log.
2. Once recovery is complete, open the database to users:
SQL> alter database open;
5.5.2.3 Oracle complete recovery from a consistent split backup image

Currently, Oracle does not support complete recovery for a database image taken using
the –consistent split option of the TimeFinder products without putting the database in
hot backup mode. Consistent split images are only supported to create database
restartable images, rather than recoverable ones. Consistent split images with the
database in hot backup mode can be used for both restart and recovery. Using EMC
consistency technology in conjunction with hot backup mode is recommended because
of the flexibility in recovery options offered.

5.5.3 Oracle incomplete recovery
Incomplete recovery procedures are nearly identical to the complete recovery steps
defined in the last section. The primary difference however, is that instead of rolling the
database to the last available transactions in the redo logs, data is only rolled forward to
an earlier point specified by the DBA. Additionally, the database must be opened using
the open resetlogs option.
Incomplete recovery is used for a number of reasons. User errors or logical corruptions
are the primary reasons that incomplete recoveries are performed (although a new
alternative to this is the Oracle Flashback technology). Another reason for performing
incomplete recovery is due to missing archive logs during a complete recovery
procedure.

5-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

5.5.3.1 Oracle incomplete recovery from a cold backup image

A point-in-time recovery of a restored image taken with the source in hot backup mode
in general requires user initiated recovery. To recover the hot backup image, follow
these steps:
1. Recover the database to a point beyond where the tablespaces are taken out of hot
backup mode:
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database until cancel;

The recover database command initiates recovery procedures within Oracle. The
control files and each datafile is examined. The database then lists the appropriate
archive logs to apply. Each archive log is then applied in turn to a point specified by
the DBA.
Alternatively, the recover database command has additional options. For example,
the database can be recovered to a specific SCN or to a particular timestamp using
the following:
SQL> recover database until change SCN;

or
SQL> recover database until time timestamp;

2. Once recovery is complete, open the database using the open resetlogs option. This
is necessary because the database was not recovered fully to the point of failure.
SQL> alter database open resetlogs;

After opening the database with the resetlogs option, you should immediately perform a
full database backup.
5.5.3.2 Oracle incomplete recovery from a hot backup image

In general, a point-in-time recovery of a restored image taken with the source in hot
backup mode requires user-initiated recovery. The following demonstrates the process
of recovering the hot backup image:
1. Recover the database to a point beyond where the tablespaces were taken out of hot
backup mode.
sqlplus “/ as sysdba”
SQL> startup mount;
SQL> recover database;

The recover database command initiates recovery procedures within Oracle. The
control files and each datafile is examined. The database then lists the appropriate
archive logs to apply. Each archive log is then applied in turn to the point all of the
tablespaces were taken out of hot backup mode.

Oracle database recovery procedures

5-19

Restoring and Recovering Oracle Databases

Alternatively, the recover database command has additional options. For example,
the database can be recovered to a specific SCN or to a particular timestamp using
the following:
SQL> recover database until change SCN;

or
SQL> recover database until time timestamp;

2. Once recovery is complete, open the database using the open resetlogs option:
SQL> alter database open resetlogs;

After opening the database with the resetlogs option, immediately perform a full
database backup.
5.5.3.3 Oracle incomplete recovery from a consistent split backup image

Currently, Oracle does not support complete recovery for a database image taken using
the –consistent split option of the TimeFinder products without putting the database in
hot backup mode. Consistent split images are only supported to create database
restartable images, rather than recoverable ones. Consistent split images with the
database in hot backup mode can be used for both restart and recovery. Using EMC
consistency technology in conjunction with hot backup mode is recommended because
of the flexibility in recovery options offered.

5.6 Database recovery using Oracle RMAN
As stated in Chapter 4, Oracle Recovery Manager provides DBAs many options when
performing recovery operations of an Oracle database backed up with the utility. The
details of how RMAN may be used to restore and recover an Oracle database are beyond
the scope of this document. The Oracle documentation Oracle Database Backup and
Recovery Basics and Oracle Database Backup and Recovery Advanced User’s Guide
provide additional detailed information on RMAN.

5.7 Oracle Flashback
Oracle Flashback is a technology that helps DBAs recover from user errors to the
database. Initial Flashback functionality was provided in Oracle9i but was greatly
enhanced in Oracle10g. Flashback retains undo data in the form of flashback logs.
Flashback logs containing undo information are periodically written by the database in
order for the various types of Flashback to work.
Each type of Flashback relies on undo data being written to the flash recovery area. The
flash recovery area is a file system Oracle uses to retain the flashback logs, archive logs,
backups, and other
Following are some of the ways Flashback helps DBAs recover from user errors:
♦ Flashback Query

5-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

♦ Flashback Version Query
♦ Flashback Transaction Query
♦ Flashback Table
♦ Flashback Drop
♦ Flashback Database
Each of these recovery methods is describe in the following sections.
Note: Flashback is a recovery mechanism for logical or user errors. It is not a utility to be used in
place of traditional backup and recovery techniques and is not designed to solve physical or
media errors.

5.7.1 Flashback configuration
Flashback is enabled in a database by creating a flash recovery area for the Flashback
logs to be retained, and by enabling Flashback logging. Flashback allows the database
to be flashed back to any point in time. However, the Flashback logs represent discrete
database points in time, and as such, ARCHIVELOG mode must also be enabled for the
database. Archive log information is used in conjunction with the flashback logs to recreate any given database point-in-time state desired.
The default flashback recovery area is defined by the Oracle initialization parameter
DB_RECOVERY_FILE_DEST. It is important to set this parameter with the location
of a directory that can hold the flashback logs. The required size of this file system
depends on how far back a user may want to flash back the database to, and whether
other objects such as archive logs and database backups, will be written to this directory.
A maximum size for this directory is specified by the
DB_RECOVERY_FILE_DEST_SIZE (no default) parameter. In some cases, Oracle
recommends up to three times the actual database size for the flash recovery area.
To enable Flashback log generation in a database, enter the following:
SQL> startup mount;
SQL> alter database flashback on;

To turn off Flashback, enter the following:
SQL> startup mount;
SQL> alter database flashback off;

To identify the state of Flashback, use the following query:
SELECT name, current_scn, flashback_on
FROM
v$database;

In addition to establishing the flash recovery area and enabling Flashback log
generation, set the initialization parameter DB_FLASHBACK_RETENTION_TARGET
(default of 1440 minutes, or one day) to define the targeted amount of Flashback log
retention. This parameter determines how far back a database can be flashed back. In

Oracle Flashback

5-21

Restoring and Recovering Oracle Databases

addition, the oldest currently available SCN and time in the Flashback logs can be
determined through the query:
SELECT oldest_flashback_scn, oldest_flashback_time
FROM
v$flashback_database_log;

Additional information concerning the flashback logs may also be found in the
v$flashback_database_log view.

5.7.2 Flashback Query
Flashback Query displays versions of queries run against a database as they looked at a
previous time. For example, if a user dropped a selection of rows from a database
erroneously, Flashback Query allows that user to run queries against the table as if it
were at that time.
The following is an example of the Flashback Query functionality:
SELECT first_name, last_name
FROM
emp
AS OF TIMESTAMP
TO_TIMESTAMP(‘2005-11-25 11:00:00’, ‘YYYY-MM-DD HH:MI:SS’)
WHERE salary = ‘100000’;

This can be used to return rows deleted as well. For example:
INSERT INTO emp
(SELECT first_name, last_name
FROM
emp
AS OF TIMESTAMP
TO_TIMESTAMP(‘2005-11-25 11:00:00’, ‘YYYY-MM-DD HH:MI:SS’)
WHERE last_name = ‘PENDLE’);

5.7.3 Flashback Version Query
Flashback Version Query displays versions of rows in a table during a specified time
interval. This functionality is helpful in auditing changes to particular rows in a
database as well as for seeing previous values of rows during a set time interval.

5.7.4 Flashback Transaction Query
Flashback Transaction Query presents changes made by transactions or sets of
transactions in the database during a specified time period.

5.7.5 Flashback Table
Flashback Table returns a table back into the state that it was at a specified time. It is
particularly useful in that this change can be made while the database is up and running.
The following is an example of the Flashback Table functionality:

5-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Restoring and Recovering Oracle Databases

FLASHBACK TABLE emp
TO TIMESTAMP
TO_TIMESTAMP(‘2005-11-26 10:30:00’, ‘YYYY-MM-DD HH:MI:SS’);

An SCN can also be used:
FLASHBACK TABLE emp
TO SCN 54395;

5.7.6 Flashback Drop
If tables in Oracle are dropped inadvertently using a DROP TABLE command,
Flashback Drop can reverse the process, reenabling access to the drop table. As long as
space is available, the DROP TABLE command does not delete data in the tablespace
data files. Instead, the table data is retained (in Oracle’s “recycle bin”) and the table is
renamed to an internally system-defined name. If the table is needed, Oracle can bring
back the table by renaming it with its old name.
The following shows an example of a table being dropped and then brought back using
the FLASHBACK TABLE command.
1. Determine the tables owned by the currently connected user:
SQL> SELECT * FROM tab;
TNAME
--------------TEST

TABTYPE
-------TABLE

CLUSTERID
---------

2. Drop the table:
SQL> DROP TABLE test;
SQL> SELECT * FROM tab;
no rows selected
SQL>

3. Ensure the table is placed in the recycle bin:
SQL> show recyclebin;
ORIGINAL NAME RECYCLEBIN NAME
OBJECT TYPE DROP TIME
------------- --------------------- ----------- -------------TEST
BIN$wdadid/3akdah3a69 TABLE
2005-11-26:10:

4. Recover the table:
FLASHBACK TABLE test
TO BEFORE DROP;

5. Verify the table is back:
SQL> SELECT * FROM tab;

Oracle Flashback

5-23

Restoring and Recovering Oracle Databases

TNAME
--------------TEST

TABTYPE
-------TABLE

CLUSTERID
---------

5.7.7 Flashback Database
Flashback Database logically recovers the entire database to a previous point in time. A
database can be rolled back in time to the point before a user error such as a batch update
or set of transactions logically corrupted the database. The database can rolled back to a
particular SCN, redo log sequence number, or timestamp. The following is the syntax of
the FLASHBACK DATABASE command:
FLASHBACK [DEVICE TYPE = device type DATABASE
TO [BEFORE] SCN = scn
TO [BEFORE] SEQUENCE = sequence # [THREAD = thread id]
TO [BEFORE] TIME = ‘date_string’

The following is an example of Flashback Database used to recover database table data
inadvertently dropped. The particular SCN before the transaction is identified and the
database flashed to an SCN just before the bad transactions occurred.
1. Identify the available Flashback logs. The following lists the available Flashback
logs and the first SCN associated with each one:
SELECT log#, first_change#, first_time
FROM v$database_flashback_logfile;
The Flashback log should be validated for the particular SCN desired.
2. Shut down the database and restart it in mount mode for the full database flashback.
SQL> shutdown immediate;
SQL> startup mount;

3. Flash back the database.
SQL> flashback database to scn = 23104;

4. Open the database for use. To make the database consistent, open the database as
follows:
SQL> alter database open resetlogs;

After opening the database with the resetlogs option, immediately perform a full
database backup.

5-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 6

Understanding
Oracle Disaster
Restart and Disaster
Recovery

This chapter presents these topics:
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6.9
6.10
6.11
6.12
6.13

Definitions .......................................................................................................... 6-2
Design considerations for disaster restart and disaster recovery ........................ 6-4
Tape-based solutions .......................................................................................... 6-8
Remote replication challenges............................................................................ 6-8
Array-based remote replication ........................................................................ 6-12
Planning for array-based replication................................................................. 6-12
SRDF/S single Symmetrix to single Symmetrix .............................................. 6-14
SRDF/S and consistency groups ...................................................................... 6-17
SRDF/A ............................................................................................................ 6-22
SRDF/AR single hop........................................................................................ 6-27
SRDF/AR multihop.......................................................................................... 6-29
Database log-shipping solutions....................................................................... 6-31
Running database solutions .............................................................................. 6-43

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

6-1

Understanding Oracle Disaster Restart and Disaster Recovery

A critical part of managing a database is planning for unexpected loss of data. The loss
can occur from a disaster such as a fire or flood or it can come from hardware or
software failures. It can even come through human error or malicious intent. In each
instance, the database must be restored to some usable point, before application services
can resume.
The effectiveness of any plan for restart or recovery involves answering the following
questions:
♦ How much downtime is acceptable to the business?
♦ How much data loss is acceptable to the business?
♦ How complex is the solution?
♦ Does the solution accommodate the data architecture?
♦ How much does the solution cost?
♦ What disasters does the solution protect against?
♦ Is there protection against logical corruption?
♦ Is there protection against physical corruption?
♦ Is the database restartable or recoverable?
♦ Can the solution be tested?
♦ If failover happens, will failback work?
All restart and recovery plans include a replication component. In its simplest form, the
replication process may be as easy as making a tape copy of the database and
application. In a more sophisticated form, it could be realtime replication of all changed
data to some remote location. Remote replication of data has its own challenges
centered around:
♦ Distance
♦ Propagation delay (latency)
♦ Network infrastructure
♦ Data loss
This section provides an introduction to the spectrum of disaster recovery and disaster
restart solutions for Oracle databases on EMC Symmetrix arrays.

6.1 Definitions
In the following sections, the terms dependent-write consistency, database restart,
database recovery, and roll-forward recovery are used. A clear definition of these terms
is required to understand the context of this section.

6-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

6.1.1 Dependent-write consistency
A dependent-write I/O is one that cannot be issued until a related predecessor I/O has
completed. Dependent-write consistency is a data state where data integrity is
guaranteed by dependent-write I/Os embedded in application logic. Database
management systems are good examples of the practice of dependent-write consistency.
Database management systems must devise protection against abnormal termination to
successfully recover from one. The most common technique used is to guarantee that a
dependent-write cannot be issued until a predecessor write has completed. Typically the
dependent-write is a data or index write while the predecessor write is a write to the log.
Because the write to the log must be completed prior to issuing the dependent-write, the
application thread is synchronous to the log write (that is, it waits for that write to
complete prior to continuing). The result of this strategy is a dependent-write consistent
database.

6.1.2 Database restart
Database restart is the implicit application of database logs during the database’s normal
initialization process to ensure a transactionally consistent data state.
If a database is shut down normally, the process of getting to a point of consistency
during restart requires minimal work. If the database abnormally terminates, then the
restart process will take longer depending on the number and size of in-flight
transactions at the time of termination. An image of the database created by using EMC
consistency technology while it is running, without conditioning the database, will be in
a dependent-write consistent data state, which is similar to that created by a local power
failure. This is also known as a DBMS restartable image. The restart of this image
transforms it to a transactionally consistent data state by completing committed
transactions and rolling back uncommitted transactions during the normal database
initialization process.

6.1.3 Database recovery
Database recovery is the process of rebuilding a database from a backup image, and then
explicitly applying subsequent logs to roll forward the data state to a designated point of
consistency. Database recovery is only possible with databases configured with archive
logging.
A recoverable Oracle database copy can be taken in one of three ways:
♦ With the database shut down and copying the database components using external
tools
♦ With the database running using the Oracle backup utility Recovery Manager
(RMAN)
♦ With the database in hot backup mode and copying the database using external
tools

Definitions

6-3

Understanding Oracle Disaster Restart and Disaster Recovery

6.1.4 Roll-forward recovery
With some databases, it may be possible to take a DBMS restartable image of the
database, and apply subsequent archive logs, to roll forward the database to a point in
time after the image was created. This means that the image created can be used in a
backup strategy in combination with archive logs. At the time of printing, a DBMS
restartable image of Oracle cannot use subsequent logs to roll forward transactions. In
most cases, during a disaster, the storage array image at the remote site will be an Oracle
DBMS restartable image and cannot have archive logs applied to it.

6.2 Design considerations for disaster restart and disaster recovery
Loss of data or loss of application availability has a varying impact from one business
type to another. For instance, the loss of transactions for a bank could cost millions,
whereas system downtime may not have a major fiscal impact. On the other hand,
businesses that are primarily web-based may require 100 percent application availability
to survive. The two factors, loss of data and loss of uptime are the business drivers that
are baseline requirements for a DR solution. When quantified, these two factors are
more formally known as Recovery Point Objective (RPO) and Recovery Time Objective
(RTO), respectively.
When evaluating a solution, the RPO and RTO requirements of the business need to be
met. In addition, the solution needs to consider operational complexity, cost, and the
ability to return the whole business to a point of consistency. Each aspect is discussed in
the following sections.

6.2.1 Recovery Point Objective
The RPO is a point of consistency to which a user wants to recover or restart. It is
measured in the amount of time from when the point of consistency was created or
captured to the time the disaster occurred. This time equates to the acceptable amount of
data loss. Zero data loss (no loss of committed transactions from the time of the
disaster) is the ideal goal, but the high cost of implementing such a solution must be
weighed against the business impact and cost of a controlled data loss.
Some organizations, like banks, have zero data loss requirements. The database
transactions entered at one location must be replicated immediately to another location.
This can have an impact on application performance when the two locations are far
apart. On the other hand, keeping the two locations close to one another might not
protect against a regional disaster like the Northeast power outage or the hurricanes in
Florida.
Defining the required RPO is usually a compromise between the needs of the business,
the cost of the solution, and the risk of a particular event happening.

6.2.2 Recovery Time Objective
The RTO is the maximum amount of time allowed for recovery or restart to a specified
point of consistency. This time involves many factors, including the time taken to:
♦ Provision power, utilities, etc.
6-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

♦ Provision servers with the application and database software.
♦ Configure the network.
♦ Restore the data at the new site.
♦ Roll forward the data to a known point of consistency.
♦ Validate the data.
Some delays can be reduced or eliminated by choosing certain DR options, such as
having a hot site where servers are preconfigured and on standby. Also, if storage-based
replication is used, the time taken to restore the data to a usable state is completely
eliminated.
As with RPO, each solution for RTO will have a different cost profile. Defining the
RTO is usually a compromise between the cost of the solution and the cost to the
business when database and applications are unavailable.

6.2.3 Operational complexity
The operational complexity of a DR solution may be the most critical factor in
determining the success or failure of a DR activity. The complexity of a DR solution
can be considered as three separate phases.
1. Initial set up of the implementation
2. Maintenance and management of the running solution
3. Execution of the DR plan in the event of a disaster
While initial configuration complexity and running complexity can be a demand on
human resources, the third phase, execution of the plan, is where automation and
simplicity must be the focus. When a disaster is declared, key personnel may be
unavailable in addition to the loss of servers, storage, networks, buildings, and so on. If
the complexity of the DR solution is such that skilled personnel with an intimate
knowledge of all systems involved are required to restore, recover and validate
application and database services, the solution has a high probability of failure.
Multiple database environments grow organically over time into complex federated
database architectures. In these federated database environments, reducing the
complexity of DR is absolutely critical. Validation of transactional consistency within
the database architecture is time consuming, costly, and requires application and
database familiarity. One reason for the complexity is the heterogeneous databases and
operating systems involved in these environments. Across multiple heterogeneous
platforms it is hard to establish a common clock and therefore hard to determine a
business point of consistency across all platforms. This business point of consistency
has to be created from intimate knowledge of the transactions and data flows.

6.2.4 Source server activity
DR solutions may or may not require additional processing activity on the source
servers. The extent of that activity can impact both response time and throughput of the

Design considerations for disaster restart and disaster recovery

6-5

Understanding Oracle Disaster Restart and Disaster Recovery

production application. This effect should be understood and quantified for any given
solution to ensure the impact to the business is minimized. The effect for some solutions
is continuous while the production application is running; for other solutions, the impact
is sporadic, where bursts of write activity are followed by periods of inactivity.

6.2.5 Production impact
Some DR solutions delay the host activity while taking actions to propagate the changed
data to another location. This action only affects write activity and although the
introduced delay may only be of the order of a few milliseconds, it can impact response
time in a high-write environment. Synchronous solutions introduce delay into write
transactions at the source site; asynchronous solutions do not.

6.2.6 Target server activity
Some DR solutions require a target server at the remote location to perform DR
operations. The server has both software and hardware costs and needs personnel with
physical access to it for basic operational functions like power on and off. Ideally, this
server could have some usage such as running development or test databases and
applications. Some DR solutions require more target server activity and some require
none.

6.2.7 Number of copies of data
DR solutions require replication of data in one form or another. Replication of a
database and associated files can be as simple as making a tape backup and shipping the
tapes to a DR site or as sophisticated as asynchronous array-based replication. Some
solutions require multiple copies of the data to support DR functions. More copies of
the data may be required to perform testing of the DR solution in addition to those that
support the DR process.

6.2.8 Distance for solution
Disasters, when they occur, have differing ranges of impact. For instance, a fire may
take out a building, an earthquake may destroy a city, or a tidal wave may devastate a
region. The level of protection for a DR solution should address the probable disasters
for a given location. For example when protecting against an earthquake, the DR site
should not be in the same locale as the production site. For regional protection, the two
sites need to be in two different regions. The distance associated with the DR solution
affects the kind of DR solution that can be implemented.

6.2.9 Bandwidth requirements
One of the largest costs for DR is in provisioning bandwidth for the solution. Bandwidth
costs are an operational expense; this makes solutions that have reduced bandwidth
requirements very attractive to customers. It is important to recognize in advance the
bandwidth consumption of a given solution to be able to anticipate the running costs.
Incorrect provisioning of bandwidth for DR solutions can have an adverse affect on
production performance and can invalidate the overall solution.

6-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

6.2.10 Federated consistency
Databases are rarely isolated islands of information with no interaction or integration
with other applications or databases. Most commonly, databases are loosely and/or
tightly coupled to other databases using triggers, database links, and stored procedures.
Some databases provide information downstream for other databases using information
distribution middleware; other databases receive feeds and inbound data from message
queues and EDI transactions. The result can be a complex interwoven architecture with
multiple interrelationships. This is referred to as a federated database architecture.
With a federated database architecture, making a DR copy of a single database without
regard to other components invites consistency issues and creates logical data integrity
problems. All components in a federated architecture need to be recovered or restarted
to the same dependent-write consistent point of time to avoid these problems.
It is possible then that point database solutions for DR, such as log shipping, do not
provide the required business point of consistency in a federated database architecture.
Federated consistency solutions guarantee that all components, databases, applications,
middleware, flat files, and such are recovered or restarted to the same dependent-write
consistent point in time.

6.2.11 Testing the solution
Tested, proven, and documented procedures are also required for a DR solution. Many
times the DR test procedures are operationally different from a set of true disaster
procedures. Operational procedures need to be clearly documented. In the best-case
scenario, companies should periodically execute the actual set of procedures for DR.
This could be costly to the business because of the application downtime required to
perform such a test, but is necessary to ensure validity of the DR solution.

6.2.12 Cost
The cost of doing DR can be justified by comparing it to the cost of not doing it. What
does it cost the business when the database and application systems are unavailable to
users? For some companies, this is easily measurable, and revenue loss can be
calculated per hour of downtime or per hour of data loss.
Whatever the business, the DR cost is going to be an extra expense item and, in many
cases, with little in return. The costs include, but are not limited to:
♦ Hardware (storage, servers and maintenance)
♦ Software licenses and maintenance
♦ Facility leasing/purchase
♦ Utilities
♦ Network infrastructure
♦ Personnel

Design considerations for disaster restart and disaster recovery

6-7

Understanding Oracle Disaster Restart and Disaster Recovery

6.3 Tape-based solutions
6.3.1 Tape-based disaster recovery
Traditionally, the most common form of disaster recovery was to make a copy of the
database onto tape and use PTAM (Pickup Truck Access Method) to take the tapes
offsite to a hardened facility. In most cases, the database and application needed to be
available to users during the backup process. Taking a backup of a running database
created a “fuzzy” image of the database on tape, one that required database recovery
after the image had been restored. Recovery usually involved application of logs that
were active during the time the backup was in process. These logs had to be archived
and kept with the backup image to ensure successful recovery.
The rapid growth of data over the last two decades indicates this method is
unmanageable. Making a hot copy of the database is now the standard, but this method
has its own challenges. How can a consistent copy of the database and supporting files
be made when they are changing throughout the duration of the backup? What exactly
is the content of the tape backup at completion? The reality is that the tape data is a
“fuzzy image” of the disk data, and considerable expertise is required to restore the
database back to a database point of consistency.
In addition, the challenge of returning the data to a business point of consistency, where
a particular database must be recovered to the same point as other databases or
applications, is making this solution less viable.

6.3.2 Tape-based disaster restart
Tape-based disaster restart is a recent development in disaster recovery strategies and is
used to avoid the “fuzziness” of a backup taken while the database and application are
running. A “restart” copy of the system data is created by locally mirroring the disks
that contain the production data, and splitting off the mirrors to create a dependent-write
consistent point-in-time image of the disks. This image is a DBMS restartable image as
described earlier. Thus, if this image was restored and the database brought up, the
database would perform an implicit recovery to attain transactional consistency. Rollforward recovery using archived logs from this database image is not possible with
Oracle without conditioning the database prior to the consistent split. This conditioning
process is described in Section 3.5.
The restartable image on the disks can be backed up to tape and moved offsite to a
secondary facility. If this image is created and shipped offsite on a daily basis, the
maximum amount of data loss is 24 hours.
The time taken to restore the database is a factor to consider since reading from tape is
typically slow. Consequently, this solution can be effective for customers with relaxed
RTOs.

6.4 Remote replication challenges
Replicating database information over long distances for the purpose of disaster
recovery is challenging. Synchronous replication over distances greater than 200 km
may be unfeasible due to the negative impact on the performance of writes because of
6-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

propagation delay; some form of asynchronous replication must be adopted.
Considerations in this section apply to all forms of remote replication technology
whether they are array-based, host-based, or managed by the database.
Remote replication solutions usually start with initially copying a full database image to
the remote location. This is called instantiation of the database. There are a variety of
ways to perform this. After instantiation, only the changes from the source site are
replicated to the target site in an effort to keep the target up to date. Some
methodologies may not send all of the changes (certain log shipping techniques for
instance), by omission rather than design. These methodologies may require periodic reinstantiation of the database at the remote site.
The following considerations apply to remote replication of databases:
♦ Propagation delay (latency due to distance)
♦ Bandwidth requirements
♦ Network infrastructure
♦ Method of instantiation
♦ Method of reinstantiation
♦ Change rate at the source site
♦ Locality of reference
♦ Expected data loss
♦ Failback operations

6.4.1 Propagation delay
Electronic operations execute at the speed of light. The speed of light in a vacuum is
186,000 miles per second. The speed of light through glass (in the case of fiber-optic
media) is less, approximately 115,000 miles per second. In other words, in an optical
network, such as SONET for instance, it takes 1 millisecond to send a data packet 125
miles or 8 milliseconds for 1,000 miles. All remote replication solutions need to be
designed with a clear understanding of the propagation delay impact.

6.4.2 Bandwidth requirements
All remote replication solutions have some bandwidth requirements because the changes
from the source site must be propagated to the target site. The more changes there are,
the greater the bandwidth that is needed. It is the change rate and replication
methodology that determine the bandwidth requirement, not necessarily the size of the
database.
Data compression can help reduce the quantity of data transmitted and therefore the size
of the “pipe” required. Certain network devices, like switches and routers, provide
native compression, some by software and some by hardware. GigE directors provide
native compression in a DMX to DMX SRDF pairing. The amount of compression

Remote replication challenges

6-9

Understanding Oracle Disaster Restart and Disaster Recovery

achieved depends on the type of data being compressed. Typical character and numeric
database data compresses at about a 2-to-1 ratio. A good way to estimate how the data
will compress is to assess how much tape space is required to store the database during a
full-backup process. Tape drives perform hardware compression on the data prior to
writing it. For instance, if a 300 GB database takes 200 GB of space on tape, the
compression ratio is 1.5 to 1.
For most customers, a major consideration in the disaster recovery design is cost. It is
important to recognize that some components of the end solution represent a capital
expenditure and some an operational expenditure. Bandwidth costs are operational
expenses and thus any reduction in this area, even at the cost of some capital expense, is
highly desirable.

6.4.3 Network infrastructure
The choice of channel extension equipment, network protocols, switches, routers, and
such, ultimately determines the operational characteristics of the solution. EMC has a
proprietary “BC Design Tool” to assist customers in analysis of the source systems and
to determine the required network infrastructure to support a remote replication solution.

6.4.4 Method of instantiation
In all remote replication solutions, a common requirement is for an initial, consistent
copy of the complete database to be replicated to the remote site. The initial copy from
source to target is called instantiation of the database at the remote site. Following
instantiation, only the changes made at the source site are replicated. For large
databases, sending only the changes after the initial copy is the only practical and costeffective solution for remote database replication.
In some solutions, instantiation of the database at the remote site uses a process similar
to the one that replicates the changes. Some solutions do not even provide for
instantiation at the remote site (log shipping for instance). In all cases it is critical to
understand the pros and cons of the complete solution.

6.4.5 Method of reinstantiation
Some methods of remote replication require periodic refreshing of the remote system
with a full copy of the database. This is called reinstantiation. Technologies such as log
shipping frequently require this since not all activity on the production database may be
represented in the log. In these cases, the disaster recovery plan must account for reinstantiation and also for the fact there may be a disaster during the refresh. The
business objectives of RPO and RTO must likewise be met under those circumstances.

6.4.6 Change rate at the source site
After instantiation of the database at the remote site, only changes to the database are
replicated remotely. There are many methods of replication to the remote site and each
has differing operational characteristics. The changes can be replicated using logging
technology (hardware and software mirroring for example). Before designing a solution
with remote replication, it is important to quantify the average change rate. It is also
important to quantify the change rate during periods of burst write activity. These

6-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

periods might correspond to end of month/quarter/year processing, billing, or payroll
cycles. The solution needs to allow for peak write workloads.

6.4.7 Locality of reference
Locality of reference is a factor that needs to be measured to understand if there will be a
reduction of bandwidth consumption when any form of asynchronous transmission is
used. Locality of reference is a measurement of how much write activity on the source
is skewed. For instance, a high locality of reference application may make many
updates to a few tables in the database, whereas a low locality of reference application
rarely updates the same rows in the same tables during a given time period. While the
activity on the tables may have a low locality of reference, the write activity into an
index might be clustered when inserted rows have the same or similar index column
values. This renders a high locality of reference on the index components.
In some asynchronous replication solutions, updates are “batched” into periods of time
and sent to the remote site to be applied. In a given batch, only the last image of a given
row/block is replicated to the remote site. So, for highly skewed application writes, this
results in bandwidth savings. Generally, the greater the time period of batched updates,
the greater the savings on bandwidth.
Log-shipping technologies do not consider locality of reference. For example, a row
updated 100 times, is transmitted 100 times to the remote site, whether the solution is
synchronous or asynchronous.

6.4.8 Expected data loss
Synchronous DR solutions are zero data loss solutions; there is no loss of committed
transactions from the time of the disaster. Synchronous solutions also may be impacted
by a rolling disaster in which case, work completed at the source site after the rolling
disaster started may be lost. Rolling disasters are discussed in detail in a later section.
Nonsynchronous DR solutions have the potential for data loss. How much data is lost
depends on many factors, most of which are defined earlier. For asynchronous
replication, where updates are batched and sent to the remote site, the maximum amount
of data lost will be two cycles or two batches worth. The two cycles that may be lost
include the cycle currently being captured on the source site and the one currently
transmitted to the remote site. With inadequate network bandwidth, data loss could
increase due the increased transmission time.

6.4.9 Failback operations
If there is the slightest chance that failover to the DR site may be required, then there is a
100 percent chance that failback to the primary site also will be required, unless the
primary site is lost permanently. The DR architecture should be designed to make
failback simple, efficient, and low risk. If failback is not planned for, there may be no
reasonable or acceptable way to move the processing from the DR site, where the
applications may be running on tier 2 servers and tier 2 networks, and so forth, back to
the production site.

Remote replication challenges

6-11

Understanding Oracle Disaster Restart and Disaster Recovery

In a perfect world, the DR process should be tested once a quarter, with database and
application services fully failed over to the DR site. The integrity of the application and
database must be verified at the remote site to ensure all required data copied
successfully. Ideally, production services are brought up at the DR site as the ultimate
test. This means production data is maintained on the DR site, requiring a failback when
the DR test completed. While this is not always possible, it is the ultimate test of a DR
solution. It not only validates the DR process, but also trains the staff on managing the
DR process should a catastrophic failure occur. The downside for this approach is that
duplicate sets of servers and storage need to be present to make an effective and
meaningful test. This tends to be an expensive proposition.

6.5 Array-based remote replication
Customers can use the capabilities of a Symmetrix storage array to replicate the database
from the production location to a secondary location. No host CPU cycles are used for
this, leaving the host dedicated to running the production application and database. In
addition, no host I/O is required to facilitate this; the array takes care of all replication
and no hosts are required at the target location to manage the target array.
EMC provides multiple solutions for remote replication of databases:
♦ SRDF/S: Synchronous SRDF
♦ SRDF/A: Asynchronous SRDF
♦ SRDF/AR: SRDF Automated Replication
Each of these solutions is discussed in detail in the following sections. To use any of the
array-based solutions, it is necessary to coordinate the disk layout of the databases with
this replication in mind.

6.6 Planning for array-based replication
All Symmetrix solutions replicating data from one array to another are disk based. This
allows the Symmetrix system to be neutral to volume manager, file system, database
system, etc. However, this does not mean that file system and volume manager concerns
can be ignored. For example, it is impossible to replicate a single disk from an AIX
volume group and import it to another host. Effectively, the smallest level of granularity
for disk-based replication is a volume group, in the case of UNIX. On Windows, the
smallest unit could be a disk, a volume set, or disk group, depending on how the disks
are set up in disk manager.
In addition, if a database is to be replicated independently of other databases, it should
have its own dedicated disks. That is, the disks used by a database should not be shared
with other applications or databases.
In many cases, when a database is restored, only the tablespace containers should be
restored and not the logs. An array-based restore copies the whole host volume, so if the
current logs need to be preserved, then they should be placed on separate volumes from
the tablespace containers. Logically, the database can be divided into recovery

6-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

structures and data. The recovery structures are those components that assist in recovery
of the database after the restore of the data components.
Figure 6-1 shows the separation of the recovery structures and data components for an
Oracle database in preparation for a TimeFinder implementation. This separation is
useful for restoring the data, and then applying the log to some known point of
consistency. This is usually for local replication and recovery purposes, but can be used
for solutions that combine database and array-based replication solutions. Typically,
this separation is less important for remote replication for restart purposes.

Figure 6-1

Database components for Oracle

When a set of volumes is defined for a database for remote replication, care must be
taken to ensure the disks hold everything needed to restart the database at the remote
site. Simply replicating the tablespace containers is in sufficient. The following is a list
of the objects that must be replicated in addition to the tablespace container directories:
♦ Oracle binaries: Remotely replicating the Oracle binaries directory is optional. If
not installed on the remote host, this directory must be replicated to start Oracle.
The version and patch level of the binaries should be the same on both the source
and target systems.
♦ Redo log directories: Place the redo log directories on the replicated disks. Use
the following series of commands to change the active log location if they are not
located on Symmetrix DMX storage:
1. Shut down the database:
sqlplus
SQL> connect / as sysdba

Planning for array-based replication

6-13

Understanding Oracle Disaster Restart and Disaster Recovery

SQL> shutdown
SQL> exit

2. Move the datafiles using O/S commands from the old location to the new
location:
mv /oracle/oldlogs/log1a.rdo /oracle/newlogs/log1a.rdo
mv /oracle/oldlogs/log1b.rdo /oracle/newlogs/log1b.rdo

3. Start the database in mount mode.
sqlplus
SQL> startup mount
SQL> alter database rename file
‘/oracle/oldlog/log1a.rdo’, ‘/oracle/newlog/log1a.rdo’
‘/oracle/oldlog/log1b.rdo’, ‘/oracle/newlog/log1b.rdo’;

4. Open the database.
SQL> alter database open;

♦ Archive log directory: Place the archive log directory on the replicated disks.
The archive log directory is identified in the init.ora startup file.
♦ Control files: Operational procedures are required to ensure that when additional
data files are added, the new versions are copied to the DR location or at least
placed on a replicated disk to guarantee the files are at the remote site in a disaster.
If an array-based solution is used, placement of these files on replicated disks
solves this problem.

6.7 SRDF/S single Symmetrix to single Symmetrix
Synchronous SRDF, or SRDF/S, is a method of replicating production data changes
from locations that are no greater than 200 km apart. Synchronous replication takes
writes that are inbound to the source Symmetrix system and copies them to the target
Symmetrix system. The write operation is not acknowledged as complete to the host
until both Symmetrix arrays have the data in cache. While the following examples
involve Symmetrix systems, the fundamentals of synchronous replication described here
are true for all synchronous replication solutions. Figure 6-2 on page 6-15 shows the
process.

6-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Figure 6-2

Synchronous replication internals

1. A write is received in the source Symmetrix cache. At this time, the host has not
received acknowledgement that the write is complete.
2. The source Symmetrix system uses SRDF/S to push the write to the target
Symmetrix system.
3. The target Symmetrix system sends an acknowledgement back to the source that the
write was received.
4. Ending status of the write is presented to the host.
These four steps cause a delay in the processing of writes as perceived by the database
on the source server. The amount of delay depends on the exact configuration of the
network, the storage, the write block size, and the distance between the two locations.
Note that reads to the source Symmetrix system are not affected by the replication.
The following steps outline the process of setting up synchronous replication using
Solutions Enabler (SYMCLI) commands.
1. Before the synchronous mode of SRDF can be established, initial instantiation of the
database is required. In other words, first create a baseline full copy of all the
volumes participating in the synchronous replication. This is usually accomplished
using the adaptive copy mode of SRDF. Create a group device_group as follows:
symdg create device_group –type rdf1

SRDF/S single Symmetrix to single Symmetrix

6-15

Understanding Oracle Disaster Restart and Disaster Recovery

2. Add disks 123, 124, and 12f to the group device_group:
symld –g device_group add dev 123
symld –g device_group add dev 124
symld –g device_group add dev 12f

3. Put the group device_group into adaptive copy mode:
symrdf –g device_group set mode acp_disk –nop

4. Instruct the source Symmetrix system to send all the tracks on the source site to the
target site using the current mode:
symrdf –g device_group establish –full -noprompt

The adaptive copy mode of SRDF has no impact to host application performance. It
transmits tracks to the remote site never sent before or changed since the last time
the track was sent. It does not preserve write order or dependent-write consistency.
5. When both sides are synchronized, put SRDF into synchronous mode. In the
following command, the device group device_group is put into synchronous mode:
symrdf –g device_group set mode sync –nop
Note: There is no requirement for a host at the remote site during the synchronous replication.
The target Symmetrix system itself manages the in-bound writes and updates the appropriate
volumes in the array.

Dependent-write consistency is inherent in a synchronous relationship as the target R2
volumes are at all times equal to the source provided that a single RA group is used. If
multiple RA groups are used or if multiple Symmetrix arrays are used on the source site,
SRDF Consistency Groups (SRDF/CG) must be used to guarantee consistency.
SRDF/CG is described below.

6.7.1 How to restart in the event of a disaster
In the event of a disaster where the primary source Symmetrix system is lost, run
database and application services from the DR site. A host at the DR site is required for
this. The first requirement is to write-enable the R2 devices. If the device_group device
group is not built on the remote host, it must be created using the R2 devices that were
mirrors of the R1 devices on the source Symmetrix system. Group Named Services
(GNS) can be used to propagate the device group to the remote site if there is a host used
there. The Solutions Enabler Symmetrix Base Management CLI Product Guide provides
more details on GNS.
To write-enable the R2s in group device_group, enter the following:
symld –g device_group rw_enable –noprompt

At this point, the host can issue the necessary commands to access the disks. For
instance, on a UNIX host, import the volume group, activate the logical volumes, fsck
the file systems and mount them.

6-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Once the data is available to the host, the database can restart.. The database will
perform an implicit recovery when restarted. Transactions that were committed, but not
completed, are rolled forward and completed using the information in the redo logs.
Transactions that have updates applied to the database, but were not committed, are
rolled back. The result is a transactionally consistent database.

6.8 SRDF/S and consistency groups
Zero data loss disaster recovery techniques tend to use straightforward database and
application restart procedures. These procedures work well if all processing and data
mirroring stop at the same instant in time at the production site, when a disaster happens.
Such is the case when there is a site power failure.
However, in most cases, it is unlikely that all data processing ceases at an instant in time.
Computing operations can be measured in nanoseconds and even if a disaster takes only
a millisecond to complete, many such computing operations could be completed
between the start of a disaster until all data processing ceases. This gives us the notion
of a rolling disaster. A rolling disaster is a series of events taken over a period of time
that comprise a true disaster. The specific period of time that makes up a rolling disaster
could be milliseconds (in the case of an explosion) or minutes in the case of a fire. In
both cases, the DR site must be protected against data inconsistency.

6.8.1 Rolling disaster
Protection against a rolling disaster is required when the data for a database resides on
more than one Symmetrix array or multiple RA groups. Figure 6-3 on page 6-18 depicts
a dependent-write I/O sequence where a predecessor log write is happening prior to a
page flush from a database buffer pool. The log device and data device are on different
Symmetrix arrays with different replication paths. Figure 6-3 on page 6-18
demonstrates how rolling disasters can affect this dependent-write sequence.

SRDF/S and consistency groups

6-17

Understanding Oracle Disaster Restart and Disaster Recovery

Figure 6-3

Rolling disaster with multiple production Symmetrix systems

1. This example of a rolling disaster starts with a loss of the synchronous links between
the bottom source Symmetrix system and the target Symmetrix system. This will
prevent the remote replication of data on the bottom source Symmetrix system.
2. The Symmetrix system, which is now no longer replicating, receives a predecessor
log write of a dependent-write I/O sequence. The local I/O is completed, however it
is not replicated to the remote Symmetrix system, and the tracks are marked as being
‘owed’ to the target Symmetrix system. Nothing prevents the predecessor log write
from completing to the host completing the acknowledgement process.
3. Now that the predecessor log writeis complete, the dependent data write is issued.
This write is received on both the source and target Symmetrix systems because the
rolling disaster does not affect those communication links.
4. If the rolling disaster ended in a complete disaster, the condition of the data at the
remote site is such that it creates a “data ahead of log” condition, which is an
inconsistent state for a database. The severity of the situation is when the database
is restarted, performing an implicit recovery, it may not detect the inconsistencies.
A person extremely familiar with the transactions running at the time of the rolling
disaster may detect the inconsistencies. Database utilities can run to detect some of
the inconsistencies.
A rolling disaster can happen so data links providing remote mirroring support are
disabled in a staggered fashion, while application and database processing continues at
the production site. The sustained replication during the time when some Symmetrix

6-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

units are communicating with their remote partners through their respective links while
other Symmetrix units are not (due to link failures) can cause data integrity exposure at
the recovery site. Some data integrity problems caused by the rolling disaster cannot be
resolved through normal database restart processing and may require a full database
recovery using appropriate backups, journals, and logs. A full database recovery
elongates overall application restart time at the recovery site.

6.8.2 Protection against a rolling disaster
SRDF consistency group (SRDF/CG) technology provides protection against rolling
disasters. A consistency group is a set of Symmetrix volumes spanning multiple RA
groups and/or multiple Symmetrix frames that replicate as a logical group to other
Symmetrix arrays using synchronous SRDF. It is not a requirement to span multiple RA
groups and/or Symmetrix frames when using consistency groups. Consistency group
technology guarantees that if a single-source volume is unable to replicate to its partner
for any reason, then all volumes in the group stop replicating. This ensures that the
image of the data on the target Symmetrix system is consistent from a dependent-write
perspective.
Figure 6-4 on page 6-19 depicts a dependent-write I/O sequence where a predecessor
log write is happening prior to a page flush from a database buffer pool. The log device
and data device are on different Symmetrix arrays with different replication paths.
Figure 6-4 demonstrates how rolling disasters can be prevented using EMC
consistencygroup technology.

Figure 6-4

Rolling disaster with SRDF consistency group protection

SRDF/S and consistency groups

6-19

Understanding Oracle Disaster Restart and Disaster Recovery

1. Consistency group protection is defined containing volumes X, Y, and Z on the
source Symmetrix system. This consistency group definition must contain all the
devices required to maintain dependent-write consistency and reside on all
participating hosts involved in issuing I/O to these devices. A mix of CKD
(mainframe) and FBA (UNIX/Windows) devices can be logically grouped together.
In some cases, the entire processing environment may be defined in a consistency
group to ensure dependent-write consistency.
2. The rolling disaster just described begins preventing the replication of changes from
volume Z to the remote site.
3. The predecessor log write occurs to volume Z, causing a consistency group
(ConGroup) trip.
4. A ConGroup trip will hold the I/O that could not be replicated along with all of the
I/O to the logically grouped devices. The I/O is held by PowerPath on the UNIX or
Windows hosts, and IOS on the mainframe host. It is held long enough to issue two
I/Os per Symmetrix system. The first I/O will put the devices in a suspend-pending
state.
5. The second I/O performs the suspend of the R1/R2 relationship for the logically
grouped devices, which immediately disables all replication to the remote site. This
allows other devices outside of the group to continue replicating provided the
communication links are available.
6. After the R1/R2 relationship is suspended, all deferred write I/Os are released,
allowing the predecessor log write to complete to the host. The dependent data write
is issued by the DBMS and arrives at X but is not replicated to the R2(X).
7. If a complete failure occurred from this rolling disaster, dependent-write consistency
at the remote site is preserved. If a complete disaster did not occur and the failed
links were activated again, the consistency group replication could be resumed once
synchronous mode is achieved. It is recommended to create a copy of the
dependent-write consistent image while the resumeoccurs. Once the SRDF process
reaches synchronization the dependent-write consistent copy is achieved at the
remote site.

6.8.3 SRDF/S with multiple source Symmetrix systems
The implications of spreading a database across multiple Symmetrix frames or across
multiple RA groups and replicating in synchronous mode were discussed in previous
sections. The challenge in this type of scenario is to protect against a rolling disaster.
SRDF consistency groups can be used to avoid data corruption in a rolling disaster
situation.
Consider the architecture depicted in Figure 6-5 on page 6-21.

6-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Figure 6-5

SRDF/S with multiple source Symmetrix systems and ConGroup protection

To protect against a rolling disaster, a consistency group can be created that
encompasses all the volumes on all Symmetrix arrays participating in replication as
shown by the blue-dotted oval.
The following steps outline the process of using Solutions Enabler (SYMCLI)
commands to set up synchronous replication with consistency groups:
1. Create a consistency group for the source side of the synchronous relationship (the
R1 side):
symcg create device_group –type rdf1 -ppath

2. Add to the consistency group the R1 devices 121 and 12f from Symmetrix with ID
111, and R1 devices 135 and 136 from Symmetrix with ID 222:
symcg
symcg
symcg
symcg

–cg
–cg
–cg
–cg

device_group
device_group
device_group
device_group

add
add
add
add

dev
dev
dev
dev

121
12f
135
136

–sid
–sid
–sid
–sid

111
111
222
222

3. Before the synchronous mode of SRDF can be established, the initial instantiation of
the database is required. In other words, first create the baseline full copy of all the
volumes participating in the synchronous replication. This is usually accomplished
using adaptive copy mode of SRDF.
4. Put the group device_group into adaptive copy mode:
symrdf –cg device_group set mode acp_disk –noprompt

SRDF/S and consistency groups

6-21

Understanding Oracle Disaster Restart and Disaster Recovery

5. Instruct the source Symmetrix system to send all tracks at the source site to the
target site using the current mode:
symrdf –cg device_group establish –full -noprompt

6. Adaptive copy mode has no host impact. It transmits tracks to the remote site never
sent before or that have changed since the last time the track was sent. It does not
preserve write order or consistency. When both sides are synchronized, SRDF can
be put into synchronous mode. In the following command, the device group proddb
is put into synchronous mode:
symrdf –cg device_group set mode sync –noprompt

7. Enable consistency protection:
symcg –cg device_group enable –noprompt
Note: There is no requirement for a host at the remote site during the synchronous replication.
The target Symmetrix system manages the in-bound writes and updates the appropriate disks in
the array.

6.9 SRDF/A
SRDF/A, or asynchronous SRDF, is a method of replicating production data changes
from one Symmetrix system to another using delta set technology. Delta sets are the
collection of changed blocks grouped together by a time interval configured at the
source site. The default time interval is 30 seconds. The delta sets are then transmitted
from the source site to the target site in the order created. SRDF/A preserves dependentwrite consistency of the database at all times at the remote site.
The distance between the source and target Symmetrix systems is unlimited and there is
no host impact. Writes are acknowledged immediately when they hit the cache of the
source Symmetrix system. SRDF/A is only available on the DMX family of Symmetrix
systems. Figure 6-6 on page 6-23 shows the process.

6-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Figure 6-6

SRDF/A replication internals

1. Writes are received in the source Symmetrix cache. The host receives immediate
acknowledgement that the write is complete. Writes are gathered into the capture
delta set for 30 seconds.
2. A delta set switch occurs and the current capture delta set becomes the transmit delta
set by changing a pointer in cache. A new empty capture delta set is created.
3. SRDF/A sends the changed blocks in the transmit delta set to the remote Symmetrix
system. The changes collect in the receive delta set at the target site. When the
replication of the transmit delta set is complete, another delta set switch occurs and a
new empty capture delta set is created with the current capture delta set becoming
the new transmit delta set. The receive delta set becomes the apply delta set.
4. The apply delta set marks all the changes in the delta set against the appropriate
volumes as invalid tracks and begins destaging the blocks to disk.
5. The cycle repeats continuously.
With sufficient bandwidth for the source database write activity, SRDF/A will transmit
all changed data within the default 30 seconds. This means that the maximum time the
target data will be behind the source is 60 seconds (two replication cycles). At times of
high-write activity, it may be impossible to transmit all the changes that occur during a
30-second interval. This means the target Symmetrix system will fall behind the source
Symmetrix system by more than 60 seconds. Careful design of the SRDF/A
infrastructure and a thorough understanding of write activity at the source site are
necessary to design a solution that meets the RPO requirements of the business at all
times.

SRDF/A

6-23

Understanding Oracle Disaster Restart and Disaster Recovery

Consistency is maintained throughout the replication process on a delta-set boundary.
The Symmetrix system will not apply a partial delta set, which would invalidate
consistency. Dependent-write consistency is preserved by placing a dependent write in
either the same delta set as the write it depends on or a subsequent delta set.
Note: There is no requirement for a host at the remote site during asynchronous replication. The
target Symmetrix system manages in-bound writes and updates the appropriate disks in the array.

Different command sets are used to enable SRDF/A depending on whether the SRDF/A
group of devices is contained within a single Symmetrix array or is spread across
multiple Symmetrix arrays.

6.9.1 SRDF/A using a single source Symmetrix system
Before the asynchronous mode of SRDF is established, initial instantiation of the
database has tooccur. In other words, a baseline full copy of all the volumes
participating in the asynchronous replication must be executed first. This is usually
accomplished using the adaptive copy mode of SRDF.
The following steps outline the process of using Solutions Enabler (SYMCLI)
commands to set up asynchronous replication.
1. Create an SRDF disk group for the source side of the synchronous relationship (the
R1 side):
symdg create device_group –type rdf1

2. Add to the device group the R1 devices 121 and 12f from the Symmetrix system
with ID 111, and R1 devices 135 and 136 from the Symmetrix system with ID 222:
symld
symld
symld
symld

–g
–g
–g
–g

device_group
device_group
device_group
device_group

add
add
add
add

dev
dev
dev
dev

121
12f
135
136

–sid
–sid
–sid
–sid

111
111
222
222

3. Put the group proddb into adaptive copy mode:
symrdf –g device_group set mode acp_disk –noprompt

4. Instruct the source Symmetrix system to send all the tracks at the source site to the
target site using the current mode:
symrdf –g device_group establish –full -noprompt

5. The adaptive copy mode of SRDF has no impact to host application performance. It
transmits tracks to the remote site never sent before or changed since the last time
the track was sent. It does not preserve write order or consistency. When both sides
are synchronized, SRDF can be put into asynchronous mode. In the following
command, the device group proddb is put into asynchronous mode:
symrdf –g device_group set mode async –noprompt

6-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Note: There is no requirement for a host at the remote site during the asynchronous replication.
The target Symmetrix system manages the in-bound writes and updates the appropriate disks in
the array.

6.9.2 SRDF/A multiple source Symmetrix systems
When a database is spread across multiple Symmetrix arrays and SRDF/A is used for
long-distance replication, separate software must be used to manage the coordination of
the delta-set boundaries between the participating Symmetrix arrays and to stop
replication if any of the volumes in the group cannot replicate for any reason. The
software must ensure that all delta-set boundaries on every participating Symmetrix
array in the configuration are coordinated to give a dependent-write consistent point-intime image of the database.
SRDF/A multisession consistency (MSC) provides consistency across multiple RA
groups and/or multiple Symmetrix arrays. MSC is available on 5671 microcode and
later with Solutions Enabler V6.0 and later. SRDF/A with MSC is supported by an
SRDF process daemon that performs cycle-switching and cache recovery operations
across all SRDF/A sessions in the group. This ensures that a dependent-write consistent
R2 copy of the database exists at the remote site at all times. A composite group must
be created using the SRDF consistency protection option (-rdf_consistency) and must
be enabled using the symcg enable command before the RDF daemon begins
monitoring and managing the MSC consistency group. The RDF process daemon must
be running on all hosts that can write to the set of SRDF/A volumes being protected. At
the time of an interruption (SRDF link failure, for instance), MSC analyzes the status of
all SRDF/A sessions and either commits the last cycle of data to the R2 target or
discards it.
The following steps outline the process using Solutions Enabler (SYMCLI) commands
to set up synchronous replication with consistency groups.
1. Create the replication composite group for the SRDF/A devices:
symcg create device_group -rdf_consistency -type rdf1

The –rdf_consistency option indicates the volumes in the group are to be protected
by MSC.
2. Add to the composite group named device_group the R1 devices 121 and 12f from
the Symmetrix system with ID 111 and R1 devices 135 and 136 from the Symmetrix
system with ID 222:
symcg
symcg
symcg
symcg

–cg
–cg
–cg
–cg

device_group
device_group
device_group
device_group

add
add
add
add

dev
dev
dev
dev

121
12f
135
136

–sid
–sid
–sid
–sid

111
111
222
222

SRDF/A

6-25

Understanding Oracle Disaster Restart and Disaster Recovery

3. Before the asynchronous mode of SRDF can be established, the initial instantiation
of the database is required. In other words, first create the baseline full copy of all
the volumes participating in the asynchronous replication. This is usually
accomplished using the adaptive copy mode of SRDF. The following command
puts the group proddb into adaptive copy mode:
symrdf –g device_group set mode acp_disk –noprompt

4. Instruct the source Symmetrix system to send all the tracks at the source site to the
target site using the current mode:
symrdf –g device_group establish –full -noprompt

5. The adaptive copy mode of SRDF has no impact on host application performance. It
transmits tracks to the remote site never sent before or changed since the last time
the track was sent. It does not preserve write order or consistency. When both sides
are synchronized, SRDF can be put into asynchronous mode. In the following
command, the device group proddb is put into asynchronous mode:
symrdf –g device_group set mode async –noprompt

6. Enable multisession consistency for the group:
symcg –cg device_group enable.
Note: There is no requirement for a host at the remote site during the asynchronous replication.
The target Symmetrix system itself manages the in-bound writes and updates the appropriate
disks in the array.

6.9.3 How to restart in the event of a disaster
In the event of a disaster when the primary source Symmetrix system is lost, run
database and application services from the DR site. A host at the DR site is required for
this. If the device_group device group is not built yet on the remote host, it must first be
created using the R2 devices that were mirrors of the R1 devices on the source
Symmetrix. The first thing that must be done is to write-enable the R2 devices.
symld –g device_group rw_enable –noprompt

R2s on a single Symmetrix

symcg –cg device_group rw_enable –noprompt

R2s on multiple Symmetrix

At this point, the host can issue the necessary commands to access the disks. For
instance, on a UNIX host, import the volume group, activate the logical volumes, fsck
the file systems, and mount them.
Once the data is available to the host, the database can be restarted. The database will
perform crash recovery when restarted. Transactions committed, but not completed, are
rolled forward and completed using the information in the redo logs. Transactions with
updates applied to the database, but not committed, are rolled back. The result is a
transactionally consistent database.

6-26

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

6.10 SRDF/AR single hop
SRDF Automated Replication, or SRDF/AR, is a continuous movement of dependentwrite consistent data to a remote site using SRDF adaptive copy mode and TimeFinder
consistent split technology. TimeFinder BCVs are used to create a dependent-write
consistent point-in-time image of the data to be replicated. The BCVs also have an R1
personality, which means that SRDF in adaptive copy mode can be used to replicate the
data from the BCVs to the target site. Since the BCVs are not changing, replication
completes in a finite length of time. The length of time for replication depends on the
size of the network “pipe” between the two locations, the distance between the two
locations, the quantity of changed data tracks, and the locality of reference of the
changed tracks. On the remote Symmetrix system, another BCV copy of the data is
made using data on the R2s. This is necessary because the next SRDF/AR iteration
replaces the R2 image in a nonordered fashion, and if a disaster were to occur while the
R2s were synchronizing, there would not be a valid copy of the data at the DR site. The
BCV copy of the data in the remote Symmetrix system is commonly called the “gold”
copy of the data. The whole process then repeats.
With SRDF/AR, there is no host impact. Writes are acknowledged immediately when
they hit the cache of the source Symmetrix system. Figure 6-7 shows the process.

Figure 6-7

SRDF/AR single-hop replication internals

1. Writes are received in the source Symmetrix cache and are acknowledged
immediately. The BCVs are already synchronized with the STDs at this point. A
consistent split is executed against the STD-BCV pairing to create a point-in-time
image of the data on the BCVs.
2. SRDF transmits the data on the BCV/R1s to the R2s in the remote Symmetrix
system.

SRDF/AR single hop

6-27

Understanding Oracle Disaster Restart and Disaster Recovery

3. When the BCV/R1 volumes are synchronized with the R2 volumes, they are
reestablished with the standards in the source Symmetrix system. This causes the
SRDF links to be suspended. At the same time, an incremental establish is
performed on the target Symmetrix system to create a “gold” copy on the BCVs in
that frame.
4. When the BCVs in the remote Symmetrix system are fully synchronized with the
R2s, they are split and the configuration is ready to begin another cycle.
5. The cycle repeats based on configuration parameters. The parameters can specify
the cycles to begin at specific times, specific intervals, or to run back to back
Cycle times for SRDF/AR are usually in the minutes to hours range. The RPO is double
the cycle time in a worst-case scenario. This may be a good fit for customers with
relaxed RPOs.
The added benefit of having a longer cycle time is that the locality of reference will
likely increase. This is because there is a much greater chance of a track being updated
more than once in a 1-hour interval than in, for example, a 30-second interval. The
increase in locality of reference shows up as reduced bandwidth requirements for the
final solution.
Before SRDF/AR starts, instantiation of the database has tooccur.. In other words, first
create a baseline full copy of all the volumes participating in the SRDF/AR replication.
This requires a full establish to the BCVs in the source array, a full SRDF establish of
the BCV/R1s to the R2s, and a full establish of the R2s to the BCVs in the target array.
There is an option to automate the initial setup of the relationship.
As with other SRDF solutions, SRDF/AR does not require a host at the DR site. The
commands to update the R2s and manage the synchronization of the BCVs in the remote
site are all managed in-band from the production site.
The Symmetrix Automated Replication UNIX and Windows Solutions Guide on
Powerlink provides more details.
Note: SRDF/AR is primarily a restartable solution. While SRDF/AR may also be used as a
recoverable solution, difficulties arise because of the need to split the archive logs separately
from the data files after taking the tablespaces out of hot backup mode. Because of this
limitation, SRDF/AR is not recommended in Oracle environments that plan on creating a
recoverable database image at the target site.

6.10.1 How to restart in the event of a disaster
In the event of a disaster, it is necessary to determine if the most current copy of the data
is located on the remote site BCVs or R2s at the remote site. Depending on when in the
replication cycle the disaster occurs, the most current version could be on either set of
disks. This determination is simple and is described in the Symmetrix Automated
Replication UNIX and Windows Solutions Guide.

6-28

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

6.11 SRDF/AR multihop
SRDF/AR multihop is an architecture that allows long-distance replication with zero
seconds of data loss through use of a bunker Symmetrix system. Production data is
replicated synchronously to the bunker Symmetrix system, which is within 200 km of
the production Symmetrix system allowing synchronous replication, but also far enough
away that potential disasters at the primary site may not affect it. Typically, the bunker
Symmetrix system is placed in a hardened computing facility.
BCVs in the bunker frame are periodically synchronized to the R2s and consistent split
in the bunker frame to provide a dependent-write consistent point-in-time image of the
data. These bunker BCVs also have an R1 personality, which means that SRDF in
adaptive copy mode can be used to replicate the data from the bunker array to the target
site. Since the BCVs are not changing, the replication can be completed in a finite
length of time. The replication time depends on the size of the “pipe” between the
bunker location and the DR location, the distance between the two locations, the
quantity of changed data, and the locality of reference of the changed data. On the
remote Symmetrix system, another BCV copy of the data is made using the R2s. This is
because the next SRDF/AR iteration replaces the R2 image, in a nonordered fashion, and
if a disaster were to occur while the R2s were synchronizing, there would not be a valid
copy of the data at the DR site. The BCV copy of the data in the remote Symmetrix
system is commonly called the “gold” copy of the data. The whole process then repeats.
With SRDF/AR multihop, there is minimal host impact. Writes are only acknowledged
when they hit the cache of the bunker Symmetrix system and a positive acknowledgment
is returned to the source Symmetrix system. Figure 6-8 depicts the process.

Figure 6-8

SRDF/AR multihop replication Internals

SRDF/AR multihop

6-29

Understanding Oracle Disaster Restart and Disaster Recovery

1. BCVs are synchronized and consistently split against the R2s in the bunker
Symmetrix system. The write activity is momentarily suspended on the source
Symmetrix system to get a dependent-write consistent point-in-time image on the
R2s in the bunker Symmetrix system, which creates a dependent-write consistent
point-in-time copy of the data on the BCVs.
2. SRDF transmits the data on the bunker BCV/R1s to the R2s in the DR Symmetrix
system.
3. When the BCV/R1 volumes are synchronized with the R2 volumes in the target
Symmetrix system, the bunker BCV/R1s are established again with the R2s in the
bunker Symmetrix system. This causes the SRDF links to be suspended between the
bunker and DR Symmetrix systems. Simultaneously, an incremental establish is
performed on the DR Symmetrix system to create a gold copy on the BCVs in that
frame.
4. When the BCVs in the DR Symmetrix system are fully synchronized with the R2s,
they are split and the configuration is ready to begin another cycle.
5. The cycle repeats based on configuration parameters. The parameters can specify
the cycles to begin at specific times, specific intervals, or to run immediately after
the previous cycle completes.
Even though cycle times for SRDF/AR multihop are usually in the minutes to hours
range, the most current data is always in the bunker Symmetrix system. Unless there is a
regional disaster that destroys both the primary site and the bunker site, the bunker
Symmetrix system will transmit all data to the remote DR site. This means zero data
loss at the point of the beginning of the rolling disaster or an RPO of 0 seconds. This
solution is a good fit for customers with a requirement of zero data loss and longdistance DR.
An added benefit of having a longer cycle time means that the locality of reference will
likely increase. This is because there is a much greater chance of a track being updated
more than once in a 1-hour interval than in say a 30-second interval. The increase in
locality of reference shows up as reduced bandwidth requirements for the network
segment between the bunker Symmetrix and the DR Symmetrix.
Before SRDF/AR can be initiated, initial instantiation of the database is required. In
other words, first create a baseline full copy of all the volumes participating in the
SRDF/AR replication. This requires a full establish of the R1s in the source location to
the R2s in the bunker Symmetrix system. The R1s and R2s need to be synchronized
continuously. The following then occur:
♦ A full establish from the R2s to the BCVs in the bunker Symmetrix system
♦ A full SRDF establish of the BCV/R1s to the R2s in the DR Symmetrix system
♦ A full establish of the R2s to the BCVs in the DR Symmetrix system are
performed.
There is an option to automate this process of instantiation.

6-30

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

The Symmetrix Automated Replication UNIX and Windows Solutions Guide provides
more information.

6.11.1 How to restart in the event of a disaster
In the event of a disaster, it is necessary to determine if the most current copy of the data
is on the R2s on the remote site or on the BCV/R1s in the bunker Symmetrix.
Depending on when the disaster occurs, the most current version could be on either set
of disks. This determination is simple and is outlined in the Symmetrix Automated
Replication UNIX and Windows Solutions Guide.

6.12 Database log-shipping solutions
Log shipping is a strategy that some companies employ for disaster recovery. The
process only works for databases using archive logging. The essence of log shipping is
that changes to the database at the source site reflected in the log are propagated to the
target site. These logs are then applied to a “standby” database at the target site to
maintain a consistent image of the database that can be used for DR purposes.

6.12.1 Overview of log shipping
The change activity on the source database generates log information eventually copied
from the redo logs to the archive logs to free up active log space. A process external to
the database takes the archived logs and transmits them (usually over IP) to a remote DR
location. This location has a database in standby mode. A server at the standby location
receives the archive logs and uses them to roll forward changes to the standby database.
If a disaster were to happen at the primary site, the standby database is brought online
and made available to users, albeit with some loss of data.

6.12.2 Log-shipping considerations
When considering a log shipping strategy it is important to understand:
♦ What log shipping covers.
♦ What log shipping does not cover.
♦ Server requirements.
♦ How to instantiate and reinstantiate the database.
♦ How failback works.
♦ Federated consistency requirements.
♦ How much data will be lost in the event of a disaster.
♦ Manageability of the solution.
♦ Scalability of the solution.

Database log-shipping solutions

6-31

Understanding Oracle Disaster Restart and Disaster Recovery

6.12.2.1 Log-shipping limitations

Log shipping transfers only the changes happening to the database that are written into
the redo logs and then copied to an archive log. Consequently, operations happening in
the database not written to the redo logs do not get shipped to the remote site. To ensure
that all transactions are written to the redo logs, run the following command:
alter database force logging;

Log shipping is a database-centric strategy. It is completely agnostic and does not
address changes that occur outside of the database. Changes include, but are not limited
to the following:
♦ Application files and binaries
♦ Database configuration files
♦ Database binaries
♦ OS changes
♦ Flat files
To sustain a working environment at the DR site, there are several procedures required
to keep these objects up to date.
6.12.2.2 Server requirements

Log shipping requires a server at the remote DR site to receive and apply the logs to the
standby database. It may be possible to offset this cost by using the server for other
functions when it is not being used for DR. Database licensing fees for the standby
database may also apply.
6.12.2.3 How to instantiate and reinstantiate the database

Log-shipping architectures need to be supported by a method of instantiating the
database at the remote site. The method needs to be manageable and timely. For
example, shipping 200 tapes from the primary site to the DR site may not be an adequate
approach, considering the transfer time and database restore time.
Reinstantiation must also be managed. Some operations, mentioned above, do not carry
over into the standby database. Periodically, it may be necessary to reinstantiate the
database at the DR site. The process should be easily managed, but also should provide
continuous DR protection. That is to say, there must be a contingency plan for a disaster
during reinstantiation.
6.12.2.4 How failback works

An important component of any DR solution is designing a failback procedure. If the
DR setup is tested with any frequency, this method should be simple and risk free. Log
shipping is done in reverse and works well when the primary site is still available. In the

6-32

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

case of a disaster where the primary site data is lost, the database is reinstantiated at the
production site.
6.12.2.5 Federated consistency requirements

Most databases are not isolated islands of information. They frequently have upstream
inputs and downstream outputs, triggers, and stored procedures that reference other
databases. There may also be a workflow management system like MQ Series, Lotus
Notes, or TIBCO managing queues containing work to be performed. This entire
environment is a federated structure that needs to be recovered to the same point in time
to get a transactionally consistent disaster restart point.
Log-shipping provides single database-centric solutions and are not adequate solutions
in federated database environments.
6.12.2.6 Data loss expectations

If sufficient bandwidth is provisioned for the solution, the amount of data lost in a
disaster is going to be approximately two logs worth of information. In terms of time, it
would be approximately twice the length of time it takes to create an archive log. This
time will most likely vary during the course of the day due to fluctuations in write
activity.
6.12.2.7 Manageability of the solution

The manageability of a DR solution is a key to its success. Log-shipping solutions have
many components to manage including servers, databases, and external objects as noted
above. Some of the questions to answer to make a clear determination of the
manageability of a log-shipping solution are:
♦ How much effort does it take to set up log shipping?
♦ How much effort is needed to keep it running on an on-going basis?
♦ What is the risk if something required at the target site is missed?
♦ If FTP is being used to ship the log files, what kind monitoring is needed to
guarantee success?
6.12.2.8 Scalability of the solution

The scalability of a solution is directly linked to its complexity. To successfully scale
the DR solution, the following questions must be answered:
♦ How much more effort does it take to add more databases?
♦ How easy is the solution to manage when the database grows much larger?
♦ What happens if the quantity of updates increases dramatically?

Database log-shipping solutions

6-33

Understanding Oracle Disaster Restart and Disaster Recovery

6.12.3 Log shipping and remote standby database
A remote standby database is an Oracle database that is a physical copy of the
production database and requires a standby control file. It can be created by restoring a
backup of the production database or by using a storage hardware mirroring technology
along with the standby controlfile.
Figure 6-9 on page 6-34 depicts a standby database kept up to date using log shipping.

Figure 6-9

Log shipping and remote standby database

1. The database must be instantiated at the DR site, either by tape or by using SRDF, or
if it is small enough, shipping it over IP.
2. As redo logs are filled, they are copied into archive logs in the archive directory.
3. A process external to the database copies the archive logs from the production site to
the DR site.
4. Periodically, the archive logs are applied to the remote standby database.
A standby database is kept in mount mode so that new logs shipped from the production
database can be applied. As new logs arrive, use the following command to apply the
logs to the standby database:
recover from archive_dir standby database;

6-34

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

In the event of a disaster, the standby database needs to be activated for production
availability:
recover database;
alter database open;

Note that users need to run a catalog for the production database to connect to the new
location or the IP address of the standby server needs to be updated to be the same as the
failed production server.

Database log-shipping solutions

6-35

Understanding Oracle Disaster Restart and Disaster Recovery

6.12.4 Log shipping and standby database with SRDF
Rather than using a file transfer protocol to ship the logs from the source to the target
site, SRDF can be used as the transfer mechanism. The advantages to using SRDF to
ship the logs are listed next.
Synchronous SRDF can be used to ship active and archive logs when the Symmetrix
arrays are less than 200 km apart; this is a zero data loss solution. When the redo logs
are replicated synchronously and a disaster occurs, all data up to the last completed
transaction will be present at the remote site. When the database is restarted at the
remote site, partial transactions will be rolled back and the committed transactions
finished using log information.
In addition, federated DR requirements can be satisfied by synchronously replicating
data external to the database.
♦ Guaranteed delivery – SRDF brings deterministic protocols into play, an important
advantage over FTP. This means that SRDF guarantees what is sent is what is
received. If packets are lost on the network, SRDF retransmits the packets as
necessary.
♦ Restartability – SRDF also brings restartability to the log shipping functionality.
Should a network outage or interruption cause data transfer to stop, SRDF can
restart from where it left off after the network is returned to normal operations.
While replicating the logs to the remote site, the receiving volumes (R2s) cannot be used
by the host, as they are read-only. If the business requirements are such that the standby
database should be continually applying logs, BCVs can be used periodically to
synchronize against the R2s and split. Then, the BCVs can be accessed by a host at the
remote site and the archive logs retrieved and applied to the standby database.

6.12.5 Data Guard
In database release Oracle8i and more significantly in the subsequent Oracle9i release of
the product, Oracle made significant changes to the standby database process described
in the previous section. Rather than utilizing host- or storage-based replication methods
for shipping the archive logs and manually applying them, Oracle created Data Guard,
which automated the replication or redo information and the application of this data to a
standby database under a single utility. Oracle9i and Oracle10g Data Guard allows
DBAs to replicate transactions written to the primary database’s redo logs, and either
asynchronously or synchronously apply them to a standby database running either in a
local or remote database environment. Oracle Data Guard provides businesses with a
simple and efficient disaster recovery mechanism for their enterprise database protection
needs.
6.12.5.1 Data Guard overview

Oracle Data Guard ensures high availability, data protection, and disaster recovery of a
database. Data Guard provides a comprehensive set of services that create, maintain,

6-36

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

manage, and monitor one or more standby databases to enable production Oracle
databases to survive disasters and data corruptions. Data Guard maintains these standby
databases as transactionally consistent copies of the production database. If the
production database becomes unavailable because of a planned or unplanned outage,
Data Guard can switch any standby database to the production role, minimizing the
downtime associated with the outage. Data Guard can be used with traditional backup,
restore, and cluster techniques to provide a high level of data protection and availability.
A Data Guard configuration consists of one production database, also referred to as the
primary database, and one or more (up to nine) standby databases. The databases in a
Data Guard configuration are connected by Oracle Net and may be dispersed
geographically. There are no restrictions on where the databases are located, provided
they can communicate with each other.
A standby database is a transactionally consistent copy of the primary database. Using a
backup copy of the primary database, up to nine standby databases can be created and
incorporated into a Data Guard configuration. Once created, Data Guard automatically
maintains each standby database by transmitting redo data from the primary database,
and then applying this redo to the standby.
6.12.5.2 Data Guard protection modes

Oracle Data Guard first became available in release Oracle8i. In that version of the
software, synchronous replication of redo log transactions is unavailable; only
asynchronous shipping of the archive logs could be configured. With Oracle8i Data
Guard, the ARCH process was responsible for replicating the redo information over the
network to the standby server. Once received by the remote host, a remote file server
(RFS) process would write the redo information to an archive log where it could be
applied to the standby database.
The Oracle9i Data Guard release substantially improved this process. In this release, the
LGWR is used to replicate redo information from the primary to standby host. Because
LGWR is used to copy the redo information, rather than the ARCH process, both
synchronous and asynchronous replication is available. Three modes of operation are
configurable in Oracle9i and Oracle10g Data Guard:
♦ Maximum Protection – This is a synchronous method of replicating redo
information. Data must be written on the standby side before acknowledgement on
the primary database. If network connectivity to the standby host is lost (that is,
the primary and standby database cannot be kept in synchronization), then the
primary database will be shut down. Network latency and bandwidth significantly
impact primary database performance.
♦ Maximum Availability – This is a synchronous method of replicating redo
information. Data must be written on the standby host before acknowledgement on
the primary database. If network connectivity to the standby host is lost however,
this mode will allow continued primary database operations. Network latency and
bandwidth significantly impact primary database performance.
♦ Maximum Performance – This is an asynchronous method of replicating redo
information. Replicated log information is allowed to diverge from the primary

Database log-shipping solutions

6-37

Understanding Oracle Disaster Restart and Disaster Recovery

database. Small amounts of data may not be replicated from the primary to standby
database, but performance impact on the primary is minimal.
While the first two options provide synchronous replication between source and target
servers, latency limitations usually prevent their use except in short-distance situations.
6.12.5.3 Data Guard services

Data Guard consists of three high-level services: Log Transport services, Log Apply
services, and Role Management services.
♦ Log Transport – Log Transport services are used to ensure that redo log
information is successfully replicated from the primary site to the target site. Log
Transport uses Oracle Net over the customers existing LAN/WAN to replicate redo
log information. Log information is transported in three forms: asynchronously as
entire archive logs (using the ARC process), asynchronously as writes to the
standby redo logs (LGWR writes to async log ship buffers), or synchronously as
writes to the standby redo logs (using the LGWR process).
♦ Log Apply – Log Apply services are configured on the standby host and are used to
read log information replicated from the primary database and apply it to the
standby. Primary database log information is read from archived logs or standby
redo logs, both located on the standby host. Two host processes are used for this:
MRP (Redo Apply) and LSP (SQL Apply).
♦ Role Management – Role Management services are used to control switchover and
failover of the primary database. During a switchover operation, the primary
database is gracefully demoted to standby status, while a standby database is
promoted to a primary without data loss. Failover operations are initiated when the
primary fails. In this case, all log information is applied and the standby database
is configured to run as the new primary database.
Figure 6-10 on page 6-39 shows a sample high-level overview of an Oracle10g Data
Guard configuration. Additional details on the role of the processes shown in the
diagram are found in the next section.

6-38

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

Figure 6-10

Sample Oracle10g Data Guard configuration

6.12.5.4 Data Guard processes

Oracle uses several old and new background processes (listed in Table 6-1) for
managing a Data Guard environment.
Table 6-1

Background processes for managing a Data Guard environment
Process

Description

LGWR
(Log Writer)

Sends Redo Log information from the primary host to the standby host
via Oracle Net. LGWR can be configured to send data to standby
redo logs on the standby host for synchronous operations.

ARCn
(Archiver)

Sends primary database archive logs to the standby host. This
process is used primarily in configurations that do not use standby
redo logs and are configured for asynchronous operations.

RFS
(Remote File Server)

Receives log data, either from the primary LGWR or ARCn processes,
and write data on the standby site to either the standby redo logs or
archive logs. This process is configured on the standby host when
Data Guard is implemented.

FAL
(Fetch Archive Log)

Manages the retrieval of corrupted or missing archive from the primary
to the standby host.

MRP
(Managed
Recovery)

Used by a physical standby database to apply logs, retrieved from
either the standby redo logs or from local copies of archive logs.

LSP
(Logical Standby)

Used by a logical standby database to apply logs, retrieved from either
the standby redo logs or from local copies of the archive logs.

LNS
(Network Server)

Enables asynchronous writes to the standby site using the LGWR
process and standby redo logs.

Database log-shipping solutions

6-39

Understanding Oracle Disaster Restart and Disaster Recovery

6.12.5.5 Physical and logical standby databases

A standby database may be configured either as a physical or a logical standby database.
A physical standby database is a block-for-block copy of the primary database. Redo
log information is read and applied by the Redo Apply process (MRP) to individual data
blocks on the standby database, similar to the way recovery would be performed if
needed on the primary database. This has a number of advantages including:
♦ A physical standby is an exact block-level copy of the primary database and is used
as a backup source for recovery directly to the primary.
♦ Data Guard only ships change records from the redo logs to the target; this
information can be significantly smaller than the actual block-level changes in the
database.
♦ Recovery time is generally short as the database is in mount mode and redo log
information is applied to the database as it is received. However, recovery times
can only be reduced if logs are continuously being applied, requiring that the
standby database always remain in managed recovery mode. While in managed
recovery mode, the standby database may not be opened in read-only (or read/write
with Oracle10g and Flashback) mode for user queries.
Some things to consider when employing a physical standby database include:
♦ A physical standby is completely dependent on redo log information. Therefore, if
any type of unlogged operation is made to the primary database (such as batch
loads, index creation, and so on) it will invalidate the standby.
♦ A physical standby only protects a single database source. If there are any external
files that require protection or a data dependency exists between the primary
database and another application, Data Guard will not protect these types of
configurations. Examples of this include environments with asynchronous
messaging, other nonOracle databases, related applications with additional data, or
configuration files that require continuous protection.
♦ A physical standby database has no mechanism to coordinate Enterprise
Consistency—the protection and recovery point between multiple databases. If
loosely or tightly coupled databases require a single point of recovery, achieving
this by means of Data Guard is operationally complex.
♦ In an environment where multiple databases exist, each requires its own Data
Guard configuration and protection. In data centers where customers have many
database instances running Data Guard, management of the database systems can
become complex. Oracle Data Guard Broker is a distributed management
framework that simplifies management in these types of environments.
A logical standby database also reads the redo log information replicated from the
primary host. However, instead of applying changes directly to the database, SQL
statements based on the redo information are generated and then run against the standby.
The SQL Apply process is used to keep the logical standby database in close

6-40

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

synchronization with the primary. Some advantages with a logical standby database
include:
♦ Read/write access – With a logical standby, the database can be open (replicated
tables are read-only) for queries or updates while the SQL Apply process is
applying data.
♦ Altering the database – Database changes can be made to a logical standby that do
not prevent additional updates from the primary. For example, additional indexes
can be created on tables to improve query performance to support a reporting
instance. Likewise, new tables or materialized views could be created allowing
read/write access or improve performance for user queries.
Additional considerations for a logical stand database are:
♦ Because a logical standby is not a physical copy of the primary database and has its
own database physical structure and ID, it cannot be used for media recovery of the
primary database. It can only maintain similar database content.
♦ Because a logical standby is not tied to the primary database structure, with the
ability to add indexes, materialized views, and other Oracle objects to improve its
usage, there is an increased chance that the standby will get out of sync with the
primary database. This makes a logical standby database a less than optimal
solution for DR.
6.12.5.6 Oracle Data Guard Broker

Oracle Enterprise Manager (OEM) provides a web-based interface for viewing,
monitoring, and administering primary and standby databases in a Data Guard
configuration. The Data Guard Broker is a distributed management framework that
automates and centralizes the creation, maintenance, and monitoring of a Data Guard
implementation. Data Guard Broker can either use the OEM GUI or a CLI to automate
and simplify:
♦ Creating and enabling Data Guard configurations, including setting up log transport
and log apply services.
♦ Managing an entire Data Guard configuration from any system in the
configuration.
In addition, the OEM GUI automates and simplifies:
♦ Creating a physical or logical standby database from a backup copy of the primary
database.
♦ Adding new or existing standby databases to an existing Data Guard configuration.
♦ Monitoring log apply rates, capturing diagnostic information, and detecting
problems quickly with centralized monitoring, testing, and performance tools.

Database log-shipping solutions

6-41

Understanding Oracle Disaster Restart and Disaster Recovery

6.12.5.7 Oracle Data Guard with EMC SRDF

SRDF and Data Guard both provide customers with the ability to create and maintain a
synchronous or asynchronous copy of their database for DR purposes at a remote site.
The decision to implement SRDF or Data Guard depends on the specific business needs
and requirements for the environment in which they are deployed. Although they
essentially perform similar tasks, there are still cases where both products may be
deployed together.
Prior to the Oracle9i release of Data Guard in which standby redo logs could be
configured, synchronous replication to the standby site could not be enabled. To ensure
no data loss between the primary and standby databases, SRDF/S can be configured in
conjunction with Data Guard to replicate only the logs as shown in Figure 6-11 on page
6-41.

Figure 6-11

“No data loss” standby database

In Oracle9i, standby redo logs were added to the database. Standby redo logs on the
target side enable Data Guard to maintain a “no data loss” standby database. These logs
act like regular redo logs but are available to receive log information synchronously
from the primary database. When the primary database flushes redo data from the log
buffers to disk, information is also sent via Oracle Net to the target site. It is received on
the target by the RFS process, which then writes it to the standby logs. This, in
conjunction with Oracle’s Real Time Apply technology in Oracle10g, enables Data
Guard to maintain a synchronous copy of the primary database at the standby site.
With the enhanced capabilities of Data Guard, in some customer environments SRDF
may be overlooked. However, even if Data Guard is planned for a particular
environment, SRDF is a useful tool for instantiating and reinstantiating an Oracle
database on the standby site. Instantiation involves creating a consistent copy of the

6-42

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Understanding Oracle Disaster Restart and Disaster Recovery

primary database at the target site. SRDF not only simplifies this process, but it is also
more efficient as incremental reestablishes may be used after the first initial full push of
the data. SRDF provides an easy-to-use and efficient mechanism for replicating the
database or outside data from the primary to standby sites, or vice versa in the event of
recovery to the primary after a failover, whenever required.

6.13 Running database solutions
6.13.1 Overview
Running database solutions attempt to use DR solutions in an active fashion. Instead of
having the database and server sitting idly waiting for a disaster to occur, the idea of
having the database running and serving a useful purpose at the DR site is an attractive
one. Also, active databases at the target site minimize the recovery time required to
have an application available in the event of a failure of the primary. The problem is
that hardware, server, and database replication-level solutions typically require exclusive
access to the database, not allowing users to access the target database. The solutions
presented in this section perform replication at the application layer and therefore allow
user access even when the database is being updated by the replication process.
In addition to an Oracle Data Guard logical standby database, which can function as a
running database while log information is being applied to it, Oracle has two other
methods of synchronizing data between disparate running databases. These running
database solutions are Oracle’s Advanced Replication and Oracle Streams, which are
described at a high level in the following sections.

6.13.2 Advanced Replication
Advanced Replication is one method of replicating objects between Oracle databases.
Advanced Replication is similar to Oracle’s previous Snapshot technology, where
changes to the underlying tables were tracked internally within Oracle and used to
provide a list of necessary rows to be sent to a remote location when a refresh of the
remote object was requested. Instead of snapshots, Oracle now uses materialized views
to track and replicate changes. Materialized views are a complete or partial copy of a
target table from a single point in time.
Advanced Replication has two types of replication sites: master sites and materialized
view sites. A master site contains information that is used as a source for the
materialized view. A materialized view site is the target site for the data to be replicated.
At the materialized view site, additional data may be written to the materialized views.
These views also may be updated with information sent back to the master site.
Materialized views with multiple master sites for a single data object are also possible.
In these situations, complexity is increased due to the need to handle conflicting data
added at each of the sites for replication to the others. This type of replication is called
Multimaster Replication.
Advanced Replication can use either asynchronous or synchronous (two-phase commit)
replication.

Running database solutions

6-43

Understanding Oracle Disaster Restart and Disaster Recovery

Advanced Replication is rarely used for DR purposes. It is typically used to replicate
infrequently changing table data between databases.
The Oracle documentation Oracle Database Advanced Replication provides more
information on Advanced Replication.

6.13.3 Oracle Streams
Streams is Oracle’s distributed transaction solution for propagating table, schema, or
entire database changes to one or many other Oracle databases. Streams uses the
concept of change records from the source database, which are used to asynchronously
distribute changes to one or more target databases. Both DML and DDL changes can be
propagated between the source and target databases. Queues on the source and target
databases are used to manage change propagation between the databases.
The process for distributing transactions includes the following stages:
♦ Capture – LCRs are created that capture DML or DDL changes from targeted
objects in the source databases.
♦ Stage – LCRs are stored in a queue to be forwarded on to the target database(s).
♦ Propagate – LCRs are the passed via the network to queues located in the target
database(s).
♦ Consume – LCRs are extracted off the queue with the corresponding DML or DDL
changes being applied to the target database(s).
A key feature of Streams is the ability to detect and resolve conflicts between the
databases participating in the replication process.
The Oracle Streams feature is rarely used for DR due to its asynchronous nature and
inherent complexity.
More detailed information on Streams is provided in the Oracle documentation Oracle
Streams Concepts and Administration and Oracle Streams Replication Administrator’s
Guide.

6-44

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 7

Oracle Database
Layouts on EMC
Symmetrix DMX

This chapter presents these topics:
7.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
7.9

The performance stack ....................................................................................... 7-2
Traditional Oracle layout recommendations ...................................................... 7-4
Symmetrix DMX performance guidelines ......................................................... 7-5
RAID considerations ........................................................................................ 7-16
Host- versus array-based striping ..................................................................... 7-21
Data placement considerations ......................................................................... 7-24
Other layout considerations.............................................................................. 7-28
Oracle database-specific configuration settings ............................................... 7-30
The database layout process ............................................................................. 7-33

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

7-1

Oracle Database Layouts on EMC Symmetrix DMX

Monitoring and managing database performance should be a continuous process in all
Oracle environments. Establishing baselines and collecting database performance
statistics for comparison against them are important to monitor performance trends and
maintain a smoothly running system. The following section discusses the performance
stack and how database performance should be managed in general. Subsequent
sections discuss Symmetrix DMX layout and configuration issues to help ensure the
database meets the required performance levels.

7.1 The performance stack
Performance tuning involves the identification and elimination of bottlenecks in the
various resources that make up the system. Resources include the application, the code
(SQL) that drives the application, the database, the host, and the storage. Tuning
performance involves the following:
♦ Analyzing each of these individual components that make up an application
♦ Identifying bottlenecks or potential optimizations that can be made to improve
performance
♦ Implementing changes that eliminate the bottlenecks or improve performance
♦ Verifying that the change has improved overall performance
Tuning performance is an iterative process and is performed until the benefits to be
gained by continued tuning are outweighed by the effort required to tune the system.
Figure 7-1 on page 7-3 shows the various “layers” to be examined as a part of any
performance analysis. The potential benefits achieved by analyzing and tuning a
particular layer of the performance stack are not equal, however. In general, tuning the
upper layers of the performance stack, such as the application and SQL statements,
provides a much better return on investment than tuning the lower layers, such as the
host or storage layers. For example, implementing a new index on a heavily used table
that changes logical access from a full table scan to index lookup with individual row
selection can vastly improve database performance if the statement is run many times
(thousands or millions) a day.
When tuning an Oracle database application, developers, DBAs, system administrators,
and storage administrators need to work together to monitor and manage the process.
Efforts should begin at the top of the stack and address application and SQL statement
tuning before moving down into the database and host-based tuning parameters. After
all of these are addressed, storage-related tuning efforts should be performed.

7-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-1

The performance stack

7.1.1 Importance of I/O avoidance
The primary goal at all levels of the performance stack is disk I/O avoidance. In theory,
an ideal database environment is one in which most I/Os are satisfied from memory
rather than going to disk to retrieve the required data. In practice however, this is
unrealistic and careful consideration of the disk I/O subsystem is necessary. Optimizing
performance of an Oracle database on an EMC Symmetrix DMX involves a detailed
evaluation of the I/O requirements of the proposed application or environment. A
thorough understanding of the performance characteristics and best practices of the
Symmetrix system, including the underlying storage components (disks, directors, and
so on) is also needed. Additionally, knowledge of complementary software products
such as EMC SRDF, EMC TimeFinder, EMC Symmetrix Optimizer, and backup
software, along with how using these products will affect the database, is important for
maximizing performance. Ensuring optimal configuration for the Oracle database
requires a holistic approach to application, host, and storage configuration planning.
Configuration considerations for host- and application-specific parameters are beyond
the scope of this document. Storage configuration considerations are covered in this
chapter.

7.1.2 Storage-system layer considerations
What is the best way to configure Oracle on EMC Symmetrix DMX storage? This is a
frequently asked question from customers. However, before recommendations are
made, a detailed understanding of the configuration and requirements for the database,

The performance stack

7-3

Oracle Database Layouts on EMC Symmetrix DMX

host(s), and storage environment is required. The principal goal for optimizing any
layout on the Symmetrix DMX is to maximize the spread of I/O across the components
of the array, reducing or eliminating any potential bottlenecks in the system. The
following sections examine the trade-offs between optimizing storage performance and
manageability for Oracle. They also discuss recommendations for laying out an Oracle
database on EMC Symmetrix DMX arrays, as well as settings for storage-related Oracle
configuration settings.

7.2 Traditional Oracle layout recommendations
Until recently, with the introduction of Automated Storage Management (ASM),
Oracle’s best practices for optimally laying out a database were focused on identifying
potential sources of contention for storage-related resources. Eliminating contention
involved understanding how the database managed the data flow process and ensuring
that concurrent or near-concurrent storage resource requests were separated onto
different physical spindles. Many of these recommendations still have value in a
Symmetrix DMX environment. Before examining other storage-based optimizations,
the next session presents a discussion of these recommendations.

7.2.1 Oracle’s optimal flexible architecture
Oracle has long recommended their Optimal Flexible Architecture (OFA) for laying out
databases on the storage. Although there is much disagreement as to whether OFA
provides an optimal storage layout, many of the recommendations continue to make
sense in Oracle environments on a Symmetrix DMX. Some of the recommendations for
performance that still generally apply include:
♦ Place redo log members (in addition to log groups) on separate hypers/spindles.
This minimizes contention for the logs as new writes come in from the database
and the old log is written to an archive log. It also isolates the sequential write and
read activity for these members from other volumes with different access methods.
♦ Redo logs and archive logs on separate hypers/spindles. This minimizes disk
contention when writing to the archive logs while reading from the previous redo
log.
♦ Separate INDEX tablespaces from their DATA counterparts. Index reads that
result in table reads are better serviced from different physical devices to minimize
disk contention and head movement.
♦ Isolate TEMP and UNDO tablespaces from DATA and INDEX information.
TEMP and UNDO typically do long sequential writes and reads. Sequential access
to data should be isolated from more random-access object types to limit head
movement and improve performance.
Replication of Oracle databases also plays a critical role in the way a database should be
laid out. To create a backup image of a database while it is open or hot, Oracle requires
that the archive logs must be replicated after the “inconsistent” data tablespaces (DATA,
INDEX, and so on) have been replicated. When using replication software such as EMC
TimeFinder or SRDF, log information must be copied after the data is replicated. This
requires that log information reside on separate hypers from the data volumes. When

7-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

configuring Oracle in a Symmetrix DMX environment the archive logs, redo logs, and
control files should be placed on separate hypervolumes from other data volumes. Also,
because it is easier to re-create a TEMP tablespace rather than replicate it (either locally
or remotely), temporary tablespaces should also be placed on their own separate
hypervolumes. A TEMP tablespace would then be re-created using a “CREATE
TEMPORARY TABLESPACE TEMP . . .” while the database is in mount mode, before
it is fully opened.
OFA provides some general recommendations for laying out an Oracle database on a
storage array. The key point with OFA, or any recommendation for optimizing the
layout, is that it is critical to understand both the type (sequential or random), size (large
or small), and quantity (low, medium or high) of I/O against the various tablespaces and
other elements (logs, control files, and so on) of the database. Without a clear
understanding of the data elements and the access patterns expected against them,
contention issues on the back-end directors or physical spindles may arise and seriously
degrade Oracle performance. Knowledge of the application, both data elements and
access patterns, is critical to ensuring high performance in the database environment.

7.2.2 Oracle layouts and replication considerations
If it is planned to use array replication technologies like TimeFinder and SRDF, it is
prudent to organize the database in such a way as to facilitate recovery. Since array
replication techniques copy volumes at the physical disk level (as seen by the host), all
datafiles for a database should be created on a set of disks dedicated to the database and
not be shared with other applications and databases. For UNIX systems, ensure that the
data files reside in a volume group dedicated to the database. Sharing with other
applications can cause unnecessary work for the array and wasted space on the target
volumes.
In addition to isolating the database to be copied onto its own dedicated volumes, the
database should be divided into two parts: the data structures and the recovery structures.
The recovery structures consist of the redo logs, the archive logs, and the control files.
The database data volumes hold the data files for all tablespaces in the database and the
ORACLE_HOME directory if desired.

7.2.3 Automated Storage Management
A new feature with Oracle10g release 1 related to Oracle data layouts is Oracle
Automated Storage Management (ASM). Using ASM reduces database layout
complexity and management in Oracle10g environments since the database itself
determines where “extents” for the database are placed on how they are managed.

7.3 Symmetrix DMX performance guidelines
Optimizing performance for Oracle in an EMC Symmetrix DMX environment is similar
to optimizing performance for all applications on the storage array. Maximizing
performance requires a clear understanding of the I/O requirements of the applications
accessing storage. The overall goal when laying out an application on disk devices in
the back-end of the Symmetrix DMX is to reduce or eliminate bottlenecks in the storage

Symmetrix DMX performance guidelines

7-5

Oracle Database Layouts on EMC Symmetrix DMX

system by spreading out the I/O across all of the array’s resources. Inside a Symmetrix
DMX array, there are several areas to consider:
♦ Front-end connections into the Symmetrix DMX – This includes the number of
connections from the host to the Symmetrix DMX that are required, and whether
front-end Fiber Channel ports will be directly connected or a SAN will be deployed
to share ports between hosts.
♦ Memory cache in the Symmetrix DMX – All host I/Os pass through cache on the
Symmetrix DMX. I/O can be adversely affected if insufficient cache is configured
in the Symmetrix DMX for the environment. Also, writes to individual
hypervolumes or to the array as a whole may be throttled when a threshold known
as the “write-pending limit” is reached.
♦ Back-end considerations – There are two sources of possible contention in the
back-end of the Symmetrix: the back-end directors and the physical spindles.
Proper layout of the data on the disks is needed to ensure satisfactory performance.

7.3.1 Front-end connectivity
Optimizing front-end connectivity requires an understanding of the number and size of
I/Os, both reads and writes, which will be sent between the hosts and the Symmetrix
DMX. There are limitations to the amount of I/O that each front-end director port, each
front-end director processor, and each front-end director board can handle. Additionally,
SAN fan-out counts (that is, the number of hosts that can be attached through a Fibre
Channel switch to a single front-end port) need to be carefully managed.
A key concern when optimizing front-end performance is determining which of the
following I/O characteristics is more important in the customer’s environment:
♦ Input/output operations per second (IOPS)
♦ Throughput (MB/s)
♦ A combination of IOPS and throughput
In OLTP database applications, where I/Os are typically small and random, IOPS is the
more important factor. In DSS applications, where transactions in general require large
sequential table or index scans, throughput is the more critical factor. In some
databases, a combination of OLTP- and DSS-like I/Os are required. Optimizing
performance in each type of environment requires tuning the host I/O size.
Figure 7-2 on page 7-7 depicts the relationships between the block size of a random read
request from the host, and both IOPS and throughput needed to fulfill that request from
the Symmetrix DMX.

7-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-2

Relationship between host block size and IOPS/throughput

The figure shows that the maximum number of IOPS is achieved using smaller block
sizes such as 4 KB (4096 bytes). For OLTP applications, where the typical Oracle
DB_BLOCK_SIZE is 4 KB or 8 KB, the Symmetrix DMX provides higher IOPS, but
decreased throughput. The opposite is also true for DSS applications. Tuning the host
to send larger I/O sizes for DSS applications can increase the overall throughput (MB/s)
from the front-end directors on the DMX. Database block sizes are generally larger (16
KB or even 32 KB) for DSS applications. Sizing the host I/O size as a power of two
multiple of the DB_BLOCK_SIZE and tuning the MULTI_BLOCK_READ_COUNT
appropriately is important for maximizing performance in a customer’s Oracle
environment.
Currently, each Fiber Channel port on the Symmetrix DMX is theoretically capable of
200 MB/s of throughput. In practice however, the throughput available per port is
significantly less and depends on the I/O size and on the shared utilization of the port
and processor on the director. Increasing the size of the I/O from the host perspective
decreases the number of IOPS that can be performed, but increases the overall
throughput (MB/s) of the port. As such, increasing the I/O block size on the host is
beneficial for overall performance in a DSS environment. Limiting total throughput to a
fraction of the theoretical maximum (100 to 120 MB/s is a good “rule of thumb”) will
ensure that enough bandwidth is available for connectivity between the Symmetrix
DMX and the host.

7.3.2 Symmetrix cache
The Symmetrix cache plays a key role in improving I/O performance in the storage
subsystem. The cache improves performance by allowing write acknowledgements to
be returned to a host when data is received in solid-state cache, rather than being fully
destaged to the physical disk drives. Additionally, reads benefit from cache when
sequential requests from the host allow follow-on reads to be prestaged in cache. The

Symmetrix DMX performance guidelines

7-7

Oracle Database Layouts on EMC Symmetrix DMX

following briefly describes how the Symmetrix cache is used for writes and reads, and
then discusses performance considerations for it.
7.3.2.1 Write operations and the Symmetrix cache

All write operations on a Symmetrix array are serviced by cache. When a write is
received by the front-end director, a cache slot must be found to service the write
operation. Since cache slots are a representation of the underlying hypervolume, if a
prior read or write operation caused the required data to already be loaded into cache, the
existing cache slot may be used to store the write I/O. If a cache slot representing the
storage area is not found, a call is made to locate a free cache slot for the write. The
write operation is moved to the cache slot and the slot is then marked write pending. At
a later point, Enginuity will destage the write to physical disk. The decision of when to
destage is based on overall system load, physical disk activity; read operations to the
physical disk, and availability of cache.
Cache is used to service the write operation to optimize the performance of the host
system. As write operations to cache are significantly faster than physical writes to disk
media, the write is reported as complete to the host operating system much earlier.
Battery backup and priority destage functions within the Symmetrix ensure that no data
loss occurs in the event of system power failure.
If the write operation to a given disk is delayed due to higher priority operations (read
activity is one such operation), the write-pending slot remains in cache for longer time
periods. Cache slots are allocated as needed to a volume for this purpose. Enginuity
calculates thresholds for allocations to limit the saturation of cache by a single
hypervolume. These limits are referred to as write-pending limits.
Cache allocations are based on a per hypervolume basis. As write-pending thresholds
are reached, additional allocations may occur, as well as reprioritization of write activity.
As a result, write operations to the physical disks may increase in priority to ensure that
excessive cache allocations do not occur. This is discussed in more detail in the next
section.
Thus, the cache enables buffering of writes and allows for a steady stream of write
activity to service the destaging of write operations from a host. In a “bursty” write
environment, this serves to even out the write activity. Should the write activity
constantly exceed the low write priority to the physical disk, Enginuity will raise the
priority of write operations to attempt to meet the write demand. Ultimately, should
write load from the host exceed the physical disk ability to write, the volume maximum
write-pending limit may be reached. In this condition, new cache slots only will be
allocated for writes to a particular volume once a currently allocated slot is freed by
destaging it to disk. This condition, if reached, may severely impact write operations to
a single hypervolume.
7.3.2.2 Read operations and the Symmetrix cache

As mentioned in the previous section, read operations typically have an elevated priority
for service from the physical disks. As user processes normally wait for an I/O
operation to complete before continuing, this is generally a good practice for storage
arrays, especially those able to satisfy write operations from cache.

7-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

When an I/O read request is detected from a host system, Enginuity sees if a
corresponding cache slot representing the storage area exists. If so, the read request may
be serviced immediately—this is considered a read hit. If the required data is not in
cache but free slots are available, the read operation must wait for a transfer from disk—
this is called a short read miss. If no cache slot exists for the read to be transferred into,
then a cache slot must be allocated and the read physically transferred into the slot – this
is referred to as a long read miss.
Although cache slots are 32 KB in size, a cache slot may contain only the requested data.
That is, if a read request is made for an 8 KB block, then only that 8 KB block will be
transferred into the cache slot, as opposed to reading the entire 32 KB track from disk.
The smallest read request unit is 4 KB (a sector).
Note: In the DMX-3, the cache slot size increases from 32 KB to 64 KB. Sectors also increase
from 4 KB to 8 KB.
7.3.2.3 Symmetrix cache performance considerations

An important performance consideration is to ensure that an appropriate amount of
cache is installed in the Symmetrix DMX. All I/O requests from hosts attached to the
array are serviced from the Symmetrix DMX cache. Symmetrix DMX cache can be
thought of as an extension to database buffering mechanisms. As such, many database
application environments can benefit from additional Symmetrix DMX cache. With
newly purchased arrays, appropriately sizing the cache is performed by the sales team,
based on the number and size of physical spindles, configuration (including number and
type of volumes), replication requirements (SRDF for example), and customer
requirements.
Cache usage can be monitored through a number of Symmetrix DMX monitoring tools.
Primary among these is ControlCenter Performance Manager (formerly known as
WorkLoad Analyzer). Performance Manager contains a number of views that analyze
Symmetrix DMX cache utilization at both the hypervolume and overall system level.
Views provide detailed information on specific component utilizations including disks,
directors (front-end and back-end), and cache utilization.
Symmetrix cache plays a key role in host I/O read and write performance. Read
performance can be improved through prefetching by the Symmetrix DMX if the reads
are sequential in nature. Enginuity algorithms detect sequential read activity and prestage reads from disk in cache before the data is requested. Write performance is also
greatly enhanced because all writes are acknowledged back to the host when they reach
cache rather than when they are written to disk. While reads from a specific
hypervolume can use as much cache as is required to satisfy host requests (assuming free
cache slots are available), the DMX limits the number of writes that can be written to a
single volume (the write-pending limit discussed earlier). Understanding the Enginuity
write-pending limits is important when planning for optimal performance.
As previously discussed, the write-pending limit is used to prevent high write rates to a
single hypervolume from consuming all of the storage array cache for its use, at the
expense of performance for reads or writes to other volumes. The write-pending limit
for each hypervolume is determined at system startup and depends on the number and
type of volumes configured and the amount of cache available. The limit is not

Symmetrix DMX performance guidelines

7-9

Oracle Database Layouts on EMC Symmetrix DMX

dependent on the actual size of each volume. The more cache available, the more write
requests that can be serviced in cache by each individual volume. While some sharing
of unused cache may occur (although this is not guaranteed), an upper limit of three
times the initial write-pending limit assigned to a volume is the maximum amount of
cache any hypervolume can acquire for changed tracks. If the maximum write-pending
limit is reached, destaging to disk must occur before new writes can come in. This
forced destaging to disk before a new write is received into cache limits writes to that
particular volume to physical disk write speeds. Forced destage of writes can
significantly reduce performance to a hypervolume should the write-pending limit be
reached. If performance problems with a particular volume are identified, an initial step
in determining the source of the problem should include verification of the number of
writes and the write-pending limit for that volume.
In addition to limits imposed at the hypervolume level, write-pending limits are imposed
at the system level. Two key cache utilization points for the DMX are reached when 40
percent and 80 percent of the cache is used for pending writes. Under normal operating
conditions, satisfying read requests from a host has greater priority than satisfying write
requests. However, when pending writes consume 40 percent of cache, the Symmetrix
DMX then prioritizes reads and writes equally. This repriorization can have a profound
affect on database performance. The degradation is even more pronounced if cache
utilization for writes reaches 80 percent. At that point, the DMX begins a forced destage
of writes to disk, with discernable performance degradation of both writes and reads. If
this threshold is reached, it is a clear indicator that reexamination of both the cache and
the total I/O on the array is needed.
Write-pending limits are also established for Symmetrix metavolumes. Metavolumes
are created by combining two or more individual hypervolumes into a single logical
device that is then presented to a host as a single logical unit (LUN). Metavolumes can
be created as concatenated or striped metavolumes. Striped metavolumes use a stripe
size of 960 KB. Concatenated metavolumes write data to the first hyper in the
metavolume (meta head) and fill it before beginning to write to the next member of the
metavolume. Write-pending limits for a metavolume are calculated on a member by
member (hypervolume) basis.
Determining the write-pending limit and current number of writes pending per
hypervolume can be done simply using SYMCLI commands.
The following SYMCLI command returns the write-pending limit for hypervolumes in a
Symmetrix system:
symcfg –v list | grep pending
Max # of system write pending slots:162901
Max # of DA write pending slots:81450
Max # of device write pending slots:4719

The exact number of cache slots available to writes for a hypervolumes varies with the
amount of cache available in the system. However, the maximum number of write
pending slots an individual hypervolume uses is up to three times the maximum number
of device write-pending slots listed above (3 * 4,719 = 14,157 write pending tracks).

7-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

The number of write-pending slots used by a host’s devices is found using the SYMCLI
command:
symstat –i 30
DEVICE

KB/sec
KB/sec % Hits %Seq Num WP
READ WRITE READ WRITE RD WRT READ Tracks

13:09:52
13:09:52 035A
0430
0431
0432
0434
0435
043A
043C
043E
043F
0440
0441
0442
0443
0444

(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not
(Not

Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible
Visible

)
)
)
)
)
)
)
)
)
)
)
)
)
)
)

0
0
0
0
0
0
0
0
0
0
0
0
13
0
0

0
0
0
0
0
0
0
0
0
0
0
0
1
0
0

0
0
0
0
0
0
0
0
0
0
0
0
66
0
0

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

N/A
100
100
82
0
0
N/A
N/A
N/A
N/A
N/A
N/A
0
100
N/A

N/A
0
0
28
100
100
N/A
N/A
N/A
N/A
N/A
N/A
100
0
N/A

N/A
2
100 2679
100 2527
0 2444
0 14157
0 14157
N/A
49
N/A
54
N/A
15
N/A
10
N/A
807
N/A
328
0
17
100 1597
N/A
4

From this, we see devices 434 and 435 have reached the device write-pending limit of
14,157. Further analysis on the cause of the excessive writes and methods of alleviating
this performance bottleneck against these devices should be made.
Alternatively, Performance Manager may be used to determine the device write-pending
limit, and whether device limits are being reached. Figure 7-3 on page 7-12 is a
Performance Manager view displaying both the device write-pending limits and device
write-pending counts for a given device, in this example Symmetrix device 055. For the
Symmetrix in this example, the write-pending slots per device was 9,776 and thus the
max write-pending limit was 29,328 slots (3 * 9776). In general, a distinct flat line in
such graphs indicates that a limit is reached.

Symmetrix DMX performance guidelines

7-11

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-3

Performance Manager graph of write-pending limit for a single hypervolume

Metavolumes can also be monitored using Performance Manager. Figure 7-4 on page 713 shows a four-member striped metavolume with the same workload as shown in the
previous figure. Each of the volumes services a proportion of the workload. Due to the
location of the file being created and the stripe depth used, one of the volumes incurred
more write activity. However, even in this case, it did not exceed the lowest of the
write-pending thresholds, let alone reach the maximal threshold limit.

7-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-4

Performance Manager graph of write-pending limit for a four-member metavolume

In the same way, the throughput and overall performance of the workload was
substantially improved. Figure 7-5 on page 7-14 shows a comparison of certain metrics
in this configuration. It is obvious this is not a fair comparison since the comparison
matches a single hypervolume against four hypers within the metavolume. However, it
does show that multiple disks can satisfy an intense workload, which clearly exceeds the
capability of a single device. It also serves to demonstrate the management and dynamic
allocation of cache resources for volumes.

Symmetrix DMX performance guidelines

7-13

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-5

Write workload for a single hyper and a striped metavolume

Note that the number of cache boards also has a minor affect on performance. When
comparing Symmetrix DMX arrays with the same amount of cache, increasing the
number of boards (for example, four cache boards with 16 GB each as opposed to two
cache boards with 32 GB each) has a small positive affect on the performance in DSS
applications. This is due to the increased number of paths between front-end directors
and cache, and has the affect of improving overall throughput. However, configuring
additional boards is only helpful in high-throughput environments such as DSS
applications. For OLTP workloads, where IOPS are more critical, additional cache
directors provide no added performance benefits. This is because the number of IOPS
per port or director is limited by the processing power of CPUs on each board.
Note: In the DMX-3, write-pending limits for individual volumes is modified. Instead of
allowing writes up to three times the initial write-pending limit, up to ~ 1/20 of the cache can be
used by any individual hypervolume.

7.3.3 Back-end considerations
Back-end considerations are typically the most important part of optimizing performance
on the Symmetrix DMX. Advances in disk technologies have not kept up with
performance increases in other parts of the storage array such as director and bandwidth
(that is, Direct Matrix versus Bus) performance. Disk-access speeds have increased by a
factor of three to seven in the last decade while other components have easily increased
one to three orders of magnitude. As such, most performance bottlenecks in the
Symmetrix DMX are attributable to physical spindle limitations.

7-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

An important consideration for back-end performance is the number of physical spindles
available to handle the anticipated I/O load. Each disk is capable of a limited number of
operations. Algorithms in the Symmetrix DMX Enginuity operating environment
optimize I/Os to the disks. Although this helps to reduce the number of reads and writes
to disk, access to disk, particularly for random reads, is still a requirement. If an
insufficient number of physical disks are available to handle the anticipated I/O
workload, performance will suffer. It is critical to determine the number of spindles
required for an Oracle database implementation based on I/O performance requirements,
and not solely on the physical space considerations.
To reduce or eliminate back-end performance issues on the Symmetrix DMX, carefully
spread access to the disks across as many back-end directors and physical spindles as
possible. EMC has long recommended for data placement of application data to “go
wide before going deep.” This means that performance is improved by spreading data
across the back-end directors and disks, rather than allocating individual applications to
specific physical spindles. Significant attention should be given to balancing the I/O on
the physical spindles. Understanding the I/O characteristics of each datafile and
separating high application I/O volumes on separate physical disks will minimize
contention and improve performance. Implementing Symmetrix Optimizer may also
help to reduce I/O contention between hypervolumes on a physical spindle. Symmetrix
Optimizer identifies I/O contention on individual hypervolumes and nondisruptively
moves one of the hypers to a new location on another disk. Symmetrix Optimizer is an
invaluable tool in helping to reduce contention on physical spindles should workload
requirements change in an environment.
Placement of data on the disks is another performance consideration. Due to the
rotational properties of disk platters, tracks on the outer parts of the disk perform better
than inner tracks. While the Symmetrix DMX Enginuity algorithms smooth out much of
this variation, small performance increases can be achieved by placing high I/O objects
on the outer parts of the disk. Of more importance, however, is minimizing the seek
times associated with the disk head moving between hypervolumes on a spindle.
Physically locating higher I/O devices together on the disks can significantly improve
performance. Disk head movement across the platters (seek time) is a large source of
latency in I/O performance. By placing higher I/O devices contiguously, disk head
movement may be reduced, increasing I/O performance of that physical spindle.

7.3.4 Additional layout considerations
A few additional factors can determine the best layout for a given hardware and software
configuration. It is important to evaluate each factor to create an optimal layout for an
Oracle database.
7.3.4.1 Host bus adapter

A host bus adapter (HBA) is a circuit board and/or integrated circuit adapter that
provides I/O processing and physical connectivity between a server and a storage device.
The connection may route through Fibre Channel switches if Fibre Channel FC-SW is
used. Because the HBA relieves the host microprocessor of both data storage and
retrieval tasks, it can improve the server's performance time. An HBA and its associated
disk subsystems are sometimes referred to as a disk channel.

Symmetrix DMX performance guidelines

7-15

Oracle Database Layouts on EMC Symmetrix DMX

HBAs can be a bottleneck if an insufficient number of them are provisioned for a given
throughput environment. When configuring Oracle systems, estimate the throughput
required and provision sufficient HBAs accordingly.
7.3.4.2 Host addressing limitations

There are also limitations on the number of LUNs that can be addressed on a host
channel. For instance, the maximum number of LUNs that can be presented on AIX is
512, while on other operating systems, the maximum number is 256.
These factors must be considered when designing the database storage infrastructure.
The final architecture will always be compromise between what is ideal and what is
economically feasible within the constraints of the implementation environment.

7.3.5 Configuration recommendations
Key recommendations for configuring the Symmetrix DMX for optimal performance
include the following:
♦ Understand the I/O – It is critical to understand the characteristics of the database
I/O including the number, type (read or write) size, location (that is, data files,
logs), and sequentiality of the I/Os. Empirical data or estimates are needed to assist
in planning.
♦ Physical spindles – The number of disk drives in the DMX should first be
determined by calculating the number of I/Os required, rather than solely based on
the physical space needs. The key is to ensure that the front-end needs of the
applications can be satisfied by the flow of data from the back end.
♦ Spread out the I/O – Both reads and writes should be spread across the physical
resources (front-end and back-end ports, physical spindles, hypervolumes) of the
DMX. This helps to prevent bottlenecks such as hitting port or spindle I/O limits,
or reaching write-pending limits on a hypervolume.
♦ Bandwidth – A key consideration when configuring connectivity between a host
and the Symmetrix DMX is the expected bandwidth required to support database
activity. This requires an understanding of the size and number of I/Os between
the host and the Symmetrix system. Connectivity considerations for both the
number of HBAs and Symmetrix front-end ports is required.

7.4 RAID considerations
For years, Oracle has recommended that all database storage be mirrored; their
philosophy of stripe and mirror everywhere (SAME) is well known in the Oracle
technical community. While laying out databases using SAME may provide optimal
performance in most circumstances, in some situations acceptable data performance
(IOPS or throughput) can be achieved by implementing more economical RAID
configurations such as RAID 5. Before discussing RAID recommendations for Oracle, a
definition of each RAID type available in the Symmetrix DMX is required.

7-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

7.4.1 Types of RAID
The following RAID configurations are available on the Symmetrix DMX:
♦ Unprotected – This configuration is not typically used in a Symmetrix DMX
environment for production volumes. BCVs and occasionally R2 devices (used as
target devices for SRDF) can be configured as unprotected volumes.
♦ RAID 1 – These are mirrored devices and are the most common RAID type in a
Symmetrix DMX. Mirrored devices require writes to both physical spindles.
However, intelligent algorithms in the Enginuity operating environment can use
both copies of the data to satisfy read requests not in the cache of the Symmetrix
DMX. RAID 1 offers optimal availability and performance, but at an increased
cost over other RAID protection options.
♦ RAID 5 – A relatively recent addition to Symmetrix data protection (Enginuity
5670+), RAID 5 stripes parity information across all volumes in the RAID group.
RAID 5 offers good performance and availability, at a decreased cost. Data is
striped using a stripe width of four tracks (128 KB on DMX-2 and 256 KB on
DMX-3). RAID 5 is configured either as RAID 5 3+1 (75% usable) or RAID 5
7+1 (87.5 percent usable) configurations. Figure 7-6 on page 7-17 shows the
configuration for 3+1 RAID 5 while Figure 7-7 on page 7-18 shows how a random
write in a RAID 5 environment is performed.

Figure 7-6

3+1 RAID 5 layout detail

♦ RAID-S – Proprietary EMC RAID configuration with parity information on a
single hypervolume. RAID-S functionality was optimized and renamed as parity
RAID in the Symmetrix DMX.

RAID considerations

7-17

Oracle Database Layouts on EMC Symmetrix DMX

♦ Parity RAID – Prior to the availability of RAID 5, parity RAID was implemented
in storage environments that required a lower cost and did not have high
performance requirements. Parity RAID uses a proprietary RAID protection
scheme with parity information being written on a single hypervolume. Parity
RAID is configured in 3+1 (75 percent usable) and 7+1 (87.5 percent usable)
configurations. Parity RAID is not recommended in current DMX configurations,
where RAID 5 should be used instead.
♦ RAID 1/0 – These are striped and mirrored devices. This configuration is only
used in mainframe environments.

Figure 7-7

Anatomy of a RAID 5 random write

The following describes the process of a random write to a RAID 5 volume:
1. A random write is received from the host and is placed into a data slot in cache to be
destaged to disk.
2. The write is destaged from cache to the physical spindle. When received, parity
information is calculated in cache on the drive by reading the old data and using an
XOR calculation with the new data.
3. The new parity information is written back to Symmetrix cache.
4. The new parity information is written to the appropriate parity location on another
physical spindle.
It is also important to note some of the optimizations implemented within Enginuity for
large sequential batch update (write) operations. As previously discussed, when random
write operations are processed, there may be a requirement to generate a background
read operation to be able to read and regenerate new parity information. With

7-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

subsequent optimizations, and when large sequential writes are generated by the host
application, Enginuity can calculate parity information based on the data written, and
then write the new parity information when the data is destaged to disk. In this way the
write penalty is removed. The size of the write operation must be at least a complete
RAID 5 stripe, since each stripe is 4 tracks (128 KB in DMX-2 and 256 KB in DMX-3),
in a RAID 5 (3+1) environment, the write must be 3 x 128 KB (384 KB) in a DMX-2
environment, or 3 x 256 KB (768 KB) in a DMX-3 environment. For a RAID 5 (7+1),
the write must be 7 x 128 KB (896 KB) in a DMX-2 environment, or 7 x 256 KB (1792
KB) in a DMX-3 environment. Thus, large sequential write operations, which may be
typical of large batch updates, may benefit from this optimization. Figure 7-8 shows an
example of this optimization.

Figure 7-8

Optimizing performance with RAID 5 sequential writes

Determining the appropriate level of RAID to configure in an environment depends on
the availability and performance requirements of the applications that will use the
Symmetrix DMX. Combinations of RAID types are configurable in the Symmetrix
DMX with some exceptions. For example, storage may be configured as a combination
of RAID 1 and 3+1 RAID 5 devices. Combinations of 3+1 and 7+1 parity or RAID 5
are currently not allowed in the same Symmetrix DMX. Likewise, mixing any types of
RAID 5 and parity RAID in the same frame is not allowed.
Until recently, RAID 1 was the predominant choice for RAID protection in Symmetrix
storage environments. RAID 1 provides maximum availability and enhanced
performance over other available RAID protections. In addition, performance
optimizations such as Symmetrix Optimizer, which reduces contention on the physical
spindles by nondisruptively migrating hypervolumes, and Dynamic Mirror Service
Policy (DMSP), which improves read performance by optimizing reads from both
mirrors, were only available with mirrored volumes, not with parity RAID devices.
While mirrored storage is still the recommended choice for RAID configurations in the

RAID considerations

7-19

Oracle Database Layouts on EMC Symmetrix DMX

Symmetrix DMX, the relatively recent addition of RAID 5 storage protection provides
customers with a reliable, economical alternative for their production storage needs.
RAID 5 storage protection became available with the 5670+ release of the Enginuity
operating environment. RAID 5 storage protection provides economic advantages over
using RAID 1, while providing high availability and performance. RAID 5 implements
the standard data striping and rotating parity across all members of the RAID group
(either 3+1 or 7+1). Additionally, Symmetrix Optimizer functionality is available with
RAID 5 in order to reduce spindle contention. RAID 5 provides customers with a
flexible data protection option for dealing with varying workloads and service-level
requirements. With the advent of RAID 5 protection, using parity RAID in Symmetrix
DMX systems is not recommended.

7.4.2 RAID recommendations
Oracle has long recommended RAID 1 over RAID 5 for database implementations. This
was largely attributed to RAID 5’s historical poor performance versus RAID 1 (due to
software implemented RAID schemes) and also due to high disk drive failure rates that
caused RAID 5 performance degradation after failures and during rebuilds. However,
disk drives and RAID 5 in general have seen significant optimizations and
improvements since Oracle initially recommended avoiding RAID 5. In the Symmetrix
DMX, Oracle databases can be deployed on RAID 5 protected disks for all but the
highest I/O performance intensive applications. Databases used for test, development,
QA, or reporting are likely candidates for using RAID 5 protected volumes.
Another potential candidate for deployment on RAID 5 storage is DSS applications. In
many DSS environments, read performance greatly outweighs the need for rapid writes.
This is because data warehouses typically perform loads off-hours or infrequently (once
a week or month); read performance in the form of database user queries is significantly
more important. Since there is no RAID penalty for RAID 5 read performance, only
write performance, these types of applications are generally good candidates for RAID 5
storage deployments. Conversely, production OLTP applications typically require small
random writes to the database, and as such, are generally more suited to RAID 1 storage.
An important consideration when deploying RAID 5 is disk failures. When disks
containing RAID 5 members fail, two primary issues arise—performance and data
availability. Performance will be affected when the RAID group operates in the
degraded mode, as the missing data must be reconstructed using parity and data
information from other members in the RAID group. Performance also will be affected
when the disk rebuild process is initiated after the failed drive is replaced or a hot spare
is activated. Potential data loss is the other important consideration when using RAID 5.
Multiple drive failures that cause the loss of multiple members of a single RAID group
result in loss of data. While the probability of such an event is small, the potential in
7+1 RAID 5 environment is much higher than that for RAID 1. As such, the probability
of data loss due to the loss of multiple members of RAID 5 group should be carefully
weighed against the benefits of using RAID 5.
The bottom line in choosing a RAID type is ensuring that the configuration meets the
needs of the customer’s environment. Considerations include the following:
♦ Read and write performance

7-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

♦ Balancing the I/O across the spindles and back-end of the Symmetrix system
♦ Tolerance for reduced application performance when a drive fails
♦ The consequences of losing data in the event of multiple disk failures
In general, EMC recommends RAID 1 for all types of customer data including Oracle
databases. However, RAID 5 configurations may be beneficial for many applications
and should be considered.

7.4.3 Symmetrix metavolumes
Individual Symmetrix hypervolumes of the same RAID type (RAID 1, RAID 5) may be
combined together to form a virtualized device called a Symmetrix metavolume.
Metavolumes are created for a number of reasons including:
♦ A desire to create devices that are greater than the largest hypervolume available
(in 5670 and 5671 Enginuity operating environments, this is currently just under 31
GB per hypervolume).
♦ To reduce the number of volumes presented down a front-end director or to an
HBA. A metavolume presented to an HBA only counts as a single LUN even
though the device may consist of a large number of individual hypers.
♦ To increase performance of a LUN by spreading I/O across more physical spindles.
There are two types of metavolumes: concatenated or striped. With concatenated
metavolumes, the individual hypers are combined to form a single volume, such that
data is written to the first hypervolume sequentially before moving to the next. Writes to
the metavolume start with the metahead and proceed on that physical until full, and then
move on to the next hypervolume. Striped metavolumes on the other hand, write data
across all members of the device. The stripe size is set at two cylinders or 960 KB.
In some cases, striped metavolumes are recommended over concatenated volumes in
Oracle database environments. The exceptions to this general rule occur in certain DSS
environments where metavolumes may obscure the sequential nature of the I/Os from
the Enginuity prefetching algorithms, or in cases where RAID 5 metavolumes are
created.

7.5 Host- versus array-based striping
Another hotly disputed issue when configuring a storage environment for maximum
performance is whether to use host-based or array-based striping in Oracle
environments. Striping of data across the physical disks is critically important to
database performance because it allows the I/O to be distributed across multiple
spindles. Although disk drive size and speeds have increased dramatically in recent
years, spindle technologies have not kept pace with host CPU and memory
improvements. Performance bottlenecks in the disk subsystem can develop if careful
attention is not paid to the data storage requirements and configuration. In general, the
discussion concerns trade-offs between performance and manageability of the storage
components.

Host- versus array-based striping

7-21

Oracle Database Layouts on EMC Symmetrix DMX

Oracle has recommended the SAME (Stripe and Mirror Everywhere) configuration for
many years. However, Oracle has never recommended where the striping should occur.
In general, the discussion concerns trade-offs between performance and manageability
of the storage components. The following presents in more depth the trade-offs when
using host-based and array-based striping.

7.5.1 Host-based striping
Host-based striping is configured through the Logical Volume Manager used on most
open-systems hosts. For example, in an HP-UX environment, striping is configured
when logical volumes are created in a volume group as shown below:
lvcreate -i 4 -I 64KB -L 1024 -n stripevol activevg

In this case, the striped volume is called stripevol (using the –n flag), is created on the
volume group activevg, is of volume size 1 GB (-L 1024), uses a stripe size of 64 KB (-I
64KB), and is striped across four physical volumes (-i 4). The specifics of striping data
at the host level are operating-system-dependent.
Two important things to consider when creating host-based striping are the number of
disks to configure in a stripe set and an appropriate stripe size. While no definitive
answer can be given that optimizes these settings for any given configuration, the
following are general guidelines to use when creating host-based stripes:
♦ Ensure that the stripe size used is a power of two multiple of the track size
configured on the Symmetrix DMX (i.e. a multiple of 32 KB on DMX-2 and 64KB
on DMX-3), the database, and host I/Os. Alignment of database blocks,
Symmetrix tracks, host I/O size, and the stripe size can have considerable impact
on database performance. Typical stripe sizes are 64 KB to 256 KB, although the
stripe size can be as high as 512 KB or even 1 MB.
♦ Multiples of 4 physical devices for the stripe width are generally recommended,
although this may be increased to 8 or 16 as required for LUN presentation or SAN
configuration restrictions as needed. Care should be taken with RAID 5
metavolumes to ensure that members do not end up on the same physical spindles
(a phenomenon known as vertical striping), as this may affect performance. In
general, RAID 5 metavolumes are not recommended.
♦ When configuring in an SRDF environment, smaller stripe sizes (32 KB for
example), particularly for the redo logs, are recommended. This is to enhance
performance in synchronous SRDF environments due to the limit of having only
one outstanding I/O per hypervolume on the link.
♦ Data alignment (along block boundaries) can play a significant role in performance,
particularly in Windows environments. Refer to operating-system-specific
documentation to learn how to align data blocks from the host along Symmetrix
DMX track boundaries.
♦ Ensure that volumes used in the same stripe set are located on different physical
spindles. Using volumes from the same physicals reduces the performance benefits
of using striping. An exception to this rule is when RAID 5 devices are used in
DSS environments.

7-22

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

7.5.2 Symmetrix-based striping (metavolumes)
An alternative to using host-based striping is to stripe at the Symmetrix DMX level.
Striping in the Symmetrix is accomplished through the use of striped metavolumes, as
discussed in the previous section. Individual hypervolumes are selected and striped
together, forming a single LUN presented through the front-end director to the host. At
the Symmetrix level, all writes to this single LUN are striped. Currently, the only stripe
size available for a metavolume is 960 KB. It is possible to create metavolumes with up
to 255 hypervolumes, although in practice metavolumes are usually created with 4 to 16
devices.

7.5.3 Striping recommendations
Determining the appropriate striping method depends on many factors. In general,
striping is a tradeoff between manageability and performance. With host-based striping,
CPU cycles are used to manage the stripes; Symmetrix metavolumes require no host
cycles to stripe the data. This small performance decrease in host-based striping is offset
by the fact that each device in a striped volume group maintains an I/O queue, thereby
increasing performance over a Symmetrix metavolume, which only has a single I/O
queue on the host. Recent tests show that striping at the host level provides somewhat
better performance than comparable Symmetrix-based striping, and is generally
recommended if performance is paramount. Host-based striping is also recommended
with environments using synchronous SRDF, since stripe sizes in the host can be tuned
to smaller increments than are currently available with Symmetrix metavolumes, thereby
increasing performance.
Management considerations generally favor Symmetrix-based metavolumes over hostbased stripes. In many environments, customers have achieved high-performance backend layouts on the Symmetrix system by allocating all of the storage as four-way striped
metavolumes. The advantage of this is any volume selected for host data is always
striped, with reduced chances for contention on any given physical spindle. Additional
storage requirements for any host volume group, since additional storage is configured
as a metavolume, also are striped. Management of added storage to an existing volume
group using host-based striping may be significantly more difficult, requiring in some
cases a full backup, reconfiguration of the volume group, and restore of the data to
successfully expand the stripe.
An alternative in Oracle environments gaining popularity recently is the combined use of
both host-based and array-based striping. Known as double striping or a plaid, this
configuration utilizes striped metavolumes in the Symmetrix array, which are then
presented to a volume group and striped at the host level. This has many advantages in
database environments where read access is small and highly random in nature. Since
I/O patterns are pseudo random, access to data is spread across a large quantity of
physical spindles, thereby decreasing the probability of contention on any given disk.
Double striping, in some cases, can interfere with data prefetching at the Symmetrix
DMX level when large, sequential data reads are predominant. This configuration may
be inappropriate for DSS workloads.
Another method of double striping the data is through the use of Symmetrix
metavolumes and RAID 5. A RAID 5 hypervolume stripes data across either four or
eight physical disks using a stripe size of four tracks (128 KB for DMX-2 or 256 KB for

Host- versus array-based striping

7-23

Oracle Database Layouts on EMC Symmetrix DMX

DMX-3). Striped metavolumes stripe data across two or more hypers using a stripe size
of two cylinders (960 KB in DMX-2 or 1920 KB in DMX-3). When using striped
metavolumes with RAID 5 devices, ensure that members do not end up on the same
physical spindles, as this will adversely affect performance. In many cases however,
double striping using this method also may affect prefetching for long, sequential reads.
As such, using striped metavolumes is generally not recommended in DSS
environments. Instead, if metavolumes are needed for LUN presentation reasons,
concatenated metavolumes on the same physical spindles are recommended.
The decision of whether to use host-based, array-based, or double striping in a storage
environment has elicited considerable fervor on all sides of the argument. While each
configuration has positive and negative factors, the important thing is to ensure that
some form of striping is used for the storage layout. The appropriate layer for disk
striping can have a significant impact on the overall performance and manageability of
the database system. Deciding which form of striping to use depends on the specific
nature and requirements of the database environment in which it is configured.
With the advent of RAID 5 data protection in the Symmetrix DMX, an additional option
of triple striping data using RAID 5, host-based striping, and metavolumes combined is
now available. However, triple striping increases data layout complexity, and in testing
has shown no performance benefits over other forms of striping. In fact, it is shown to
be detrimental to performance and as such, is not recommended in any Symmetrix DMX
configuration.

7.6 Data placement considerations
Placement of the data on the physical spindles can potentially have a significant impact
on Oracle database performance. Placement factors that affect database performance
include:
♦ Hypervolume selection for specific database files on the physical spindles
themselves.
♦ The spread of database files across the spindles to minimize contention.
♦ The placement of high I/O devices contiguously on the spindles to minimize head
movement (seek time).
♦ The spread of files across the spindles and back-end directors to reduce component
bottlenecks.
Each of these factors is discussed next.

7.6.1 Disk performance considerations
As shown in Figure 7-9 on page 7-26, there are five main considerations for spindle
performance:
♦ Actuator Positioning (Seek Time) – The time it takes the actuating mechanism to
move the heads from their present position to a new position. This delay averages
a few milliseconds in length and depends on the type of drive. For example, a 15k

7-24

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

drive has an average seek time of approximately 3.5 ms for reads and 4 ms for
writes, with a full disk seek of 7.4 ms for reads and 7.9 ms for writes.
Note: Disk drive characteristics can be found at www.seagate.com.

♦ Rotational Speed – This is due to the need for the platter to rotate underneath the
head to correctly position the data to be accessed. Rotational speeds for spindles in
the Symmetrix DMX range from 7,200-15,000 rpm. The average rotational delay
is the time it takes for half of a revolution of the disk. In the case of a 15 KB drive,
this is about 2 milliseconds.
♦ Interface Speed –A measure of the transfer rate from the drive into the Symmetrix
cache. It is important to ensure that the transfer rate between the drive and cache is
greater than the drive’s rate to deliver data. Delay caused by this is typically a very
small value, on the order of a fraction of a millisecond.
♦ Areal Density –A measure of the number of bits of data that fits on a given surface
area on the disk. The greater the density, the more data per second that can be read
from the disk as it passes under the disk head.
♦ Cache Capacity and Algorithms – Newer disk drives have improved read and write
algorithms, as well as cache, in order to improve the transfer of data in and out of
the drive and to make parity calculations for RAID 5.
Delay caused by the movement of the disk head across the platter surface is called seek
time. The time associated with a data track rotating to the required location under the
disk head is referred to as rotational delay. The cache capacity on the drive, disk
algorithms, interface speed, and the areal density (or zoned bit recording) combines to
produce a disk transfer time. Therefore, the time taken to complete an I/O (or disk
latency) consists of these three elements: seek time, rotational delay, and transfer time.
Data transfer times are typically on the order of fractions of a millisecond and as such,
rotational delays and delays due to repositioning the actuator heads are the primary
sources of latency on a physical spindle. Additionally, rotational speeds of disk drives
have increased from top speeds of 7,200 rpm up to 15,000 rpm, but still average on the
order of a few milliseconds. The seek time continues to be the largest source of latency
in disk assemblies when using the entire disk.
Transfer delays are lengthened in the inner parts of the drive; more data can be read per
second on the outer parts of the drive than by data located on the inner regions.
Therefore, performance is significantly improved on the outer parts of the disk. In many
cases, performance improvements of more than 50 percent can sometimes be realized on
the outer cylinders of a physical spindle. This performance differential typically leads
customers to place high I/O objects on the outer portions of the drive.
While placing high I/O objects such as redo logs on the outer edges of the spindles has
merit, performance differences across the drives inside the Symmetrix DMX are
significantly smaller than the stand-alone disk characteristics would attest. Enginuity
operating environment algorithms, particularly the algorithms that optimize ordering of
I/O as the disk heads scan across the disk, greatly reduces differences in hypervolume
performance across the drive. Although this smoothing of disk latency may actually
increase the delay of a particular I/O, overall performance characteristics of I/Os to
hypervolumes across the face of the spindle will be more uniform.

Data placement considerations

7-25

Oracle Database Layouts on EMC Symmetrix DMX

Figure 7-9

Disk performance factors

7.6.2 Hypervolume contention
Disk drives can receive only a limited number of read or write I/Os before performance
degradation occurs. While disk improvements and cache, both on the physical drives
and in disk arrays, have improved disk read and write performance, the physical devices
can still become a critical bottleneck in Oracle database environments. Eliminating
contention on the physical spindles is a key factor in ensuring maximum Oracle
performance on Symmetrix DMX arrays.
Contention can occur on a physical spindle when I/O (read or write) to one or more
hypervolumes exceeds the I/O capacity of the disk. While contention on a physical
spindle is undesirable, this type of contention can be rectified by migrating high I/O data
onto other devices with lower utilization. This is accomplished using a number of
methods, depending on the type of contention that is found. For example, when two or
more hypervolumes on the same physical spindle have excessive I/O, contention may be
eliminated by migrating one of the hypervolumes to another, lower-utilized physical
spindle. This is done through processes such as LVM mirroring at the host level or by
using tools such as EMC Symmetrix Optimizer to nondisruptively migrate data from
impacted devices. One method of reducing hypervolume contention is careful layout of
the data across the physical spindles on the back-end of the Symmetrix system. Another

7-26

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

method of reducing contention is to use striping, either at the host level or inside the
Symmetrix system.
Hypervolume contention can be found in several ways. Oracle-specific data collection
and analysis tools such as the Automatic Workload Repository (AWS), the Automatic
Database Diagnostic Monitor, and StatsPack can help to identify areas of reduced I/O
performance in the database data files. Additionally, EMC tools such as Performance
Manager (formerly WorkLoad Analyzer) can help to identify performance bottlenecks in
the Symmetrix DMX array. Establishing baselines of the system and proactive
monitoring are essential in helping to maintain an efficient, high-performance database.
Commonly, tuning database performance on the Symmetrix system is performed postimplementation. This is unfortunate because with a small amount of up-front effort and
detailed planning, significant I/O contention issues could be minimized or eliminated in
a new implementation. While detailed I/O patterns of a database environment are not
always well known, particularly in the case of a new system implementation, careful
layout consideration of a database on the back end of the Symmetrix system can save
time and future effort in trying to identify and eliminate I/O contention on the disk
drives.

7.6.3 Maximizing data spread across the back end
A long-standing data layout recommendation at EMC is “Go wide before going deep.”
This means that data placement on the Symmetrix DMX should be spread across the
back-end directors and physical spindles before locating data on the same physical
drives. By spreading the I/O across the Symmetrix back end, I/O bottlenecks in any one
array component can be minimized or eliminated.
Given recent improvements in the Symmetrix DMX component technologies, such as
CPU performance on the directors and the Direct Matrix architecture, the most common
bottleneck in new implementations is with contention on the physical spindles and the
back-end directors. To reduce these contention issues, examine the I/O requirements for
each application that will use the Symmetrix storage. From this analysis, create a
detailed layout that balances the anticipated I/O requirements across both back-end
directors and physical spindles.
Before data is laid out on the DMX back end, it is helpful to understand the I/O
requirements for each of the file systems or volumes being laid out. Several methods to
optimize layout on the back-end directors and spindles are available. One timeconsuming method involves creating a map of the hypervolumes on physical storage,
including hypervolume presentation by director and physical spindle, based on
information available in EMC ControlCenter. This involves documenting the
environment using a tool such as Excel, with each hypervolume marked on its physical
spindle and disk director. Using this map of the back end and volume information for
the database elements, preferably categorized by I/O requirement (high/medium/low, or
by anticipated reads and writes), the physical data elements and I/Os can be evenly
spread across the directors and physical spindles.
This type of layout can be extremely complex and time-consuming. Additional
complexity is added when RAID 5 hypers are added to the configuration. Since each
hypervolume is placed on either four or eight physical volumes in RAID 5

Data placement considerations

7-27

Oracle Database Layouts on EMC Symmetrix DMX

environments, trying to uniquely map out each datafile or database element is beyond
what most customers feel provides value. In these cases, one alternative is to rank each
of the database elements or volumes in terms of anticipated I/O. Once ranked, each
element may be assigned a hypervolume in order on the back end. Since BIN file
creation tools almost always spread contiguous hypervolume numbers across different
elements of the back end, this method of assigning the ranked database elements usually
provides a reasonable spread of I/O across the spindles and back-end directors in the
Symmetrix DMX. In combination with Symmetrix Optimizer, this method of spreading
the I/O is normally effective in maximizing the spread of I/O across DMX components.

7.6.4 Minimizing disk head movement
Perhaps the key performance consideration a customer can control when laying out a
database on the Symmetrix DMX is minimizing head movement on the physical
spindles. Positioning high I/O hypervolumes contiguously on the physical spindles can
minimize head movement. Disk latency caused by interface or rotational speeds cannot
be controlled by layout considerations. The only disk drive performance considerations
that can be controlled are the placement of data onto specific, higher-performing areas of
the drive (discussed in a previous section) and the reduction of actuator movement, by
trying to place high I/O objects in adjacent hypervolumes on the physical spindles.
One method, described in the previous section, describes how volumes can be ranked by
anticipated I/O requirements. Using a documented “map” of the back-end spindles,
high-I/O objects can be placed on the physical spindles, grouping the highest-I/O objects
together. Recommendations differ as to whether it is optimal to place the highest I/O
objects together on the outer parts of the spindle (the highest performing parts of a
physical spindle) or in the center of a spindle. Since there is no definitive answer to this
question, the historical recommendation of putting high-I/O objects together on the outer
part of the spindle is still a reasonable suggestion. Placing these high-I/O objects
together on the outer parts of the spindle should help reduce disk actuator movement
when doing reads and writes to each hypervolume on the spindle, thereby improving a
controllable parameter in any data layout exercise.

7.7 Other layout considerations
In addition to the layout considerations described in previous sections, a few additional
factors may be important to DBAs or storage administrators seeking to optimize
database performance. Additional configuration factors to consider include the
following:
♦ Implementing SRDF/S for the database
♦ Creating database clones using TimeFinder/Mirror or TimeFinder/Clone
♦ Creating database clones using TimeFinder/Snap
These additional layout considerations are discussed in the following sections.

7.7.1 Database layout considerations with SRDF/S
Two primary concerns must be considered when SRDF/S is implemented:

7-28

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

♦ Inherent latency is added for each write to the database. Latency occurs because
each write must be first written to both the local and remote Symmetrix caches
before the write can be acknowledged to the host. This latency must always be
considered as a part of any SRDF/S implementation. As the speed of light cannot
be circumvented, there is little to be done to mitigate this latency.
♦ Each hypervolume configured in the Symmetrix is only allowed to send a single
I/O across the SRDF link. Performance degradation results when multiple I/Os are
written to a single hypervolume since subsequent writes must wait for predecessors
to complete. Striping at the host level is particularly helpful in these situations.
Using a smaller stripe size (32-128 KB) ensures that larger writes will be spread
across multiple hypervolumes, reducing the chances for SRDF to serialize writes
across the link.

7.7.2 Database cloning, TimeFinder, and sharing spindles
Database cloning is useful when DBAs want to create backup or other business
continuance images of a database. A common question when laying out a database is
whether BCVs should share the same physical spindles as the production volumes or
should be isolated on separate physical disks. There are pros and cons to each of the
solutions; the optimal solution generally depends on the anticipated workload.
The primary benefit of spreading BCVs across all physical spindles is performance.
Spreading I/Os across more spindles reduces the risk of bottlenecks on the physical
disks. Workloads that use BCVs, such as backups and reporting databases, may
generate high I/O rates. Spreading this workload across more physical spindles may
significantly improve performance in these environments.
The main drawbacks to spreading BCVs across all spindles in the Symmetrix system
are:
♦ Synchronization may cause spindle contention during resynchronization.
♦ BCV workloads may negatively impact production database performance.
When resynchronizing the BCVs, data is read from the production hypers and copied
into cache. From there it is destaged to the BCVs. When the physical disks share
production and BCVs, the synchronization rates can be greatly reduced because of
increased seek times due to the conflict between reading from one part of the disk and
writing to another. The other drawback to sharing physical disks is the increased
workload on the spindles that may impact performance on the production volumes.
Sharing the spindles increases the chance that contention may arise, decreasing database
performance.
Determining the appropriate location for BCVs (either sharing the same physical
spindles or isolated on their own disks) depends on customer preference and workload.
In general, BCVs should share the same physical spindles. However, in cases where the
BCV synchronization and utilization may negatively impact applications (for example,
databases that run 24x7 with high I/O requirements), it may be beneficial for the BCVs
to be isolated on their own physical disks.

Other layout considerations

7-29

Oracle Database Layouts on EMC Symmetrix DMX

7.7.3 Database clones using TimeFinder/Snap
TimeFinder/Snap provides many of the benefits of full-volume replication techniques
such as TimeFinder/Mirror or TimeFinder/Clone, but at greatly reduced costs. However,
there are two performance considerations when using TimeFinder/Snap to make
database clones for backups or other business continuous functions. These include Copy
on First Write (COFW) and Copy On Access (COA) penalties. The first affects the
production volume performance, while the second affects access to the snap volumes.
COFW is the result of the need for data to be copied from the production hypers to the
save area as writes come into them. It affects the production devices. TimeFinder/Snap
uses virtual devices that contain pointers to where valid data for the snap device is
located. When first created, all of the pointers are directed at the production
hypervolume. As time goes by and changes are made to the production volume, any
changes must be saved to an alternative location before being written to disk. This
requirement to save the original data to a save device before a write can be processed,
along with the change to the snap pointer, manifests itself as a small write performance
hit to the production volumes. Although generally small, whenever a snap device is
created with a production volume, consider this COFW performance penalty.
The COA penalty affects both the production and snap volume performance. It is the
result of two reasons: The need to determine where the snap data on disk is located and
the workload on the production volumes. The virtual device contains pointers that
define whether a requested track is located on the production volume or on a save
device. This in-memory lookup before reading from the appropriate disk track requires
extra cycles of processing before a read request is returned to a host.
In addition, this COA penalty depends on the load in the Symmetrix system and the
activity. In highly utilized systems, the performance penalty can increase dramatically
due to contention for physical resources. In highly utilized systems, using
TimeFinder/Snap can result in unsatisfactory performance in all applications resident of
the array.

7.8 Oracle database-specific configuration settings
Oracle provides significant flexibility when configuring a database through its
initialization parameters. Although a broad range of parameters can be adjusted,
relatively few have a significant impact on Oracle performance from a storage
perspective.

7-30

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

Table 7-1 on page 7-32 describes initialization parameters that can be tuned to improve
I/O performance from a Symmetrix DMX storage array.

Oracle database-specific configuration settings

7-31

Oracle Database Layouts on EMC Symmetrix DMX

Table 7-1

7-32

Initialization parameters
Parameter

Description

DB_BLOCK_BUFFERS

Specifies the number of data “pages” available in host
memory for data pulled from disk. Typically, the more
block buffers available in memory, the better the
potential performance of the database.

DB_BLOCK_SIZE

Determines the size of the data pages Oracle stores in
memory and out on disk. For DSS applications, using
larger block sizes such as 16 KB (or 32 KB when
available) improves data throughput, while for OLTP
applications, a 4 KB or 8 KB block size may be more
appropriate.

DB_FILE_MULTIBLOCK_READ_
COUNT

Specifies the maximum number of blocks that can be
read in a single sequential read I/O. For OLTP
environments, this parameter should be set to a low
value (4 or 8 for example). For DSS environments
where long, sequential data scans are normal, this
parameter should be increased to match the maximum
host I/O size (or more) to optimize throughput.

DB_WRITER_PROCESSES

Specifies the number of DBWR processes initially
started for the database. Increasing the number of
DBWR processes can improve writes to disk through
multiplexing if multiple CPUs are available in the host.

DBWR_IO_SLAVES

Configures multiple I/O server process for the DBW0
process. This parameter is only used on single CPU
servers where only a single DBWR process is enabled.
Configuring I/O slaves can improve write performance
to disk through multiplexing the writes.

DISK_ASYNCH_IO

Controls whether I/O to Oracle structures such as datafiles, log files, and control files is written asynchronously
or not. If asynchronous I/O is available on the host
platform, asynchronous I/O to the datafiles has a
positive affect on I/O performance.

FAST_START_MTTR_TARGET

Specifies the desired number of seconds needed to
crash recover a database in the event of a failure. If
used, setting this to low values is detrimental to
performance because of the need to perform frequent
checkpoints to ensure quick restart of the data.

LOG_BUFFER

Specifies the size of the redo log buffer. Increasing the
size of this buffer can decrease the frequency of
required writes to disk.

LOG_CHECKPOINT_INTERVAL

Specifies the number of redo log blocks that can be
written before a checkpoint must be performed. This
affects performance since a checkpoint requires that
data be written to disk to ensure consistency. Frequent
checkpoints reduce the amount of recovery needed if a
crash occurs but can also be detrimental to Oracle
performance.

LOG_CHECKPOINT_TIMEOUT

Specifies the number of seconds that can elapse before
a checkpoint must be performed. This affects
performance since a checkpoint requires that data be
written to disk to ensure consistency. Frequent
checkpoints reduce the amount of recovery needed if a
crash occurs, but also can be detrimental to Oracle
performance.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

SORT_AREA_SIZE

Specifies the maximum amount in memory that Oracle
will use to perform sort operations. Increasing this
parameter decreases the likelihood that a sort will be
performed in a temporary tablespace on disk. However,
this also increases the memory requirements on the
host.

7.9 The database layout process
After discussing the various considerations for optimizing Oracle performance on the
Symmetrix DMX, the question arises as to how these recommendations are applied
when laying out a new database. There are four general phases to the layout process:
♦ Analysis and planning
♦ Initial database layout
♦ Implementation
♦ Reanalyze and refine
The following sections describe each of these phases and provide examples of how a
database might be laid out for three examples: an OLTP-like database, a DSS-like
database, and a mixed-type database.

7.9.1 Database layout process
As discussed previously, there are four primary phases to a database layout process. The
following describes the high-level steps involved in an Oracle database layout process.
7.9.1.1 Analysis and planning

The first step in the process is typically the most time-consuming as it requires
thoroughly analyzing and documenting the anticipated environment. Information
concerning the host, storage, connectivity, I/O, availability, growth, and backups must
be provided as a part of this phase of the process. Typical questions include:
♦ What is the anticipated size of the database at deployment? After one month?
Aftesix 6 months?
♦ What is the host environment needed for the application? Memory? CPUs?
Growth?
♦ What level of operating system will be deployed? Which LVM? Raw or “cooked”
file systems?
♦ How will data striping be achieved (host-based, storage-based, double striping)?
♦ What RAID will be used for the database environment (RAID 1, RAID 5)?
♦ How many data files for the database will be created? Which have the highest I/O
activity?

The database layout process

7-33

Oracle Database Layouts on EMC Symmetrix DMX

♦ What are the availability requirements for the database?
♦ Will a cluster be deployed? How many nodes? Single instance? RAC?
♦ How many data paths are required from the host to the storage array? Will
multipathing software be used?
♦ How will the host connect d to the storage array (direct attach, SAN)?
♦ Which is more important: IOPS or throughput? How much of each are anticipated?
♦ What kind of database is planned (DSS, OLTP, a combination of the two)?
♦ What types of I/Os are anticipated from the database (long sequential reads, small
bursts of write activity, a mix of reads and writes)?
♦ How will backups be handled? Will replication (host or storage based) be used?
Answers to these questions determines the configuration and layout of the proposed
database. The key to the layout process is a complete understanding of the database
characteristics and requirements to implement. Of particular importance when planning
a database layout is the I/O characteristics of the various database objects. This
information is collected and documented, and encompasses the key deliverable for the
next phase of the database layout on a Symmetrix project.
In some cases, the databases to be deployed already exist in a production environment.
In these cases, it is easy to understand the I/O characteristics of the various underlying
database structures (tablespaces, data files, tables, and so on). Various tools for
gathering performance statistics include Oracle StatsPack, EMC Performance Manager,
host-based utilities including sar and iostat, and third-party analyzers (such as Quest
Central). Performance statistics such as reads and writes are determined for database
objects. These statistics are then used to determine the required number of physical
spindles, the number of I/O paths between the host and the storage array, and the
Symmetrix configuration.
Sometimes, the database to be deployed is not in a production environment. In these
cases, it is difficult to anticipate the I/O requirements of the planned database. However,
EMC has analyzed empirical data from a wide variety of environments, including Oracle
database implementations, and has created workload simulations based upon these
analyses. Given an anticipated maximum number of I/Os and type of workload (DSS or
OLTP), a simulated workload can be put into an EMC utility called SymmMerge to
estimate resource utilization on the Symmetrix system. This tool is only available to
EMC internal performance specialists (i.e., SPEED resources), but using this tool
ensures a successful Oracle implementation in a Symmetrix environment when the exact
workload requirements are unknown.
7.9.1.2 Initial database layout

The next step in the process is to create an initial layout of the database on the back-end
storage of the Symmetrix. Of primary concern is spreading out the anticipated workload
across the back-end directors and physical spindles inside the Symmetrix array. The
first step in the process is to acquire a map of the Symmetrix back end. This map shows

7-34

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Oracle Database Layouts on EMC Symmetrix DMX

the layout of the Symmetrix hypervolumes on the physical disks in the array. This
information can be acquired from the EMC software Enterprise Control Center (ECC).
In addition to the back-end layout, the database configuration in terms of tablespaces and
datafiles should be planned. This requires determining the number and type of datafiles
needed for the implementation. Additionally, estimates of the type, size, and number of
read and write requests for each volume should be determined. These estimates form the
basis of the data layout process.
Once the information on the Symmetrix back-end layout and Oracle datafile
requirements are determined, the next step is to begin laying out each of the datafiles
across the hypervolumes in the Symmetrix system. Volumes are laid out such that the
workload is spread across the back-end directors and the physical spindles. Make sure
that the number of reads and writes to a physical device does not exceed the maximum
number of I/Os that a spindle can handle. However, if sufficient diligence is taken to
ensure balance of I/Os across the drives, reaching I/O limits on one spindle would result
in high rates of activity across all drives. In such a situation, additional spindles are
required to handle the workload.
Simpler alternatives to a detailed database layout exist. One method is to simply rank
I/Os (primarily reads since writes are written to cache). In a typical Symmetrix BIN file,
consecutive hypervolume numbers are spread across different physical spindles and
back-end directors. By ranking volume requirements for the database and assigning
them to consecutive hypervolume numbers, a DBA can reasonably spread the database
I/O across back-end directors and the physical spindles.
7.9.1.3 Implementation

The implementation phase takes the planned database layout from the preceding step and
implements it into the customer’s environment. The host is presented with the
documented storage. Volume groups and file systems are created as required and
database elements are initialized as planned. This phase of the process is normally short
and relatively straight-forward to complete if prior steps are performed and documented
well.
7.9.1.4 Reanalyze and refine

After the databaseis put into production, it is important to establish performance
baselines and continue to monitor the system environment. Important tools for this
include Oracle Statspack and EMC Performance Manager. Performance baselines after
deployment help to determine the database performance characteristics of the system.
Thereafter, ongoing monitoring of the system and comparison to these benchmarks helps
determine whether performance degrades over time and where the degradation occurs.
Degradation can occur in database performance due to growth in the system or changing
requirements inside the database.
If degradation is detected, there are several ways to deal with it. If the source of poor
performance is contention on the physical spindle (for example, multiple active
hypervolumes on the same physical spindle contending for I/O), then workload on the
drive must be reduced. There are several ways to do this. Striping data is effective at
eliminating contention on a spindle. Another commonly used method to eliminate

The database layout process

7-35

Oracle Database Layouts on EMC Symmetrix DMX

contention is to migrate one of the active hypervolumes on the drive to a new location.
This can be done through host-based mechanisms such as copying the associated data
files to new hypervolumes. However, this can cause service disruptions to the data
being migrated.
An alternative to host-based migrations is to use EMC Symmetrix Optimizer.
Symmetrix Optimizer proactively analyzes the Symmetrix for back-end performance
issues including director and disk contention. It then determines the best way to
optimize disk I/O. It detects hypervolume contention on a physical spindle and migrates
the volume to a new location without disruption to active systems.
Another source of degradation occurs when activity to a single hypervolume exceeds
hypervolume (write pending) or spindle (read and write) limits. In such cases, moving
the hypervolume to another physical spindle will not solve the problem. When this case
is found, the only way to reduce the performance degradation is to spread out the
contents of the hypervolume onto multiple volumes. If multiple datafiles are located on
the hyper, migrating one or more to alternate locations may fix the problem. More
difficult (and more common however) is when a single datafile his issue is to have the
data spread across multiple hypervolumes. This can be done through striping techniques
at the Symmetrix system through metavolumes, at the host through host-based striping,
or in the database by using data partitioning.

7-36

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Chapter 8

Data Protection

This chapter presents these topics:
8.1
8.2
8.3
8.4

EMC Double Checksum overview..................................................................... 8-2
Implementing EMC Double Checksum for Oracle ............................................ 8-3
Implementing Generic SafeWrite for generic applications ................................ 8-6
Syntax and examples ........................................................................................ 8-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

8-1

Data Protection

This chapter describes data protection methods using EMC Double Checksum to
minimize the impact of I/O errors on database consistency during I/O transfers between
hosts and Symmetrix storage devices.

8.1 EMC Double Checksum overview
The EMC Double Checksum feature provides a method to help minimize the impact of
I/O errors on database consistency during I/O transfers between hosts and Symmetrix
storage devices.
For Oracle, EMC Double Checksum for Oracle contains a rich set of checks that can be
natively performed by the Symmetrix. For each Relational Database Management
System (RDBMS) write in the Symmetrix system, checksum values are computed and
compared to test the data for any corruption picked up along the way from the host.
Although errors of this kind are infrequent, they can have a considerable effect on data
availability and recovery. Section 8.2 provides details on this feature.
For generic RDBMS applications, EMC Double Checksum for Generic Applications
provides the Generic SafeWrite1 feature to help protect critical applications from
incurring an incomplete write, and subsequent torn page, due to a failure with a
component connected to the Symmetrix Front End Channel Adapter. Generic SafeWrite
is most often used to protect against corruptions due to HBA and link failures including
server crashes, where essentially, it will help protect against fractured writes that can
occur before the data reaches the Symmetrix. Section 8.3 provides details on this feature.
This chapter contains overview and concept information. The EMC Solutions Enabler
Symmetrix CLI Command Reference contains a complete list of syntax and options.

8.1.1 Traditional methods of preventing data corruption
Data corruption checking is an integral part of most RDBMS products. For instance,
Oracle computes a checksum that verifies the data within each page. If corruption
occurs, the checksum will be incorrect when the data is read. However, this checking
only takes place within the host system – not the storage array.
As a result, if there is corruption after the data leaves the host system, it will not be
detected until that data is read back into the system, which can be some time — maybe
months — later. The RDBMS will issue an alert, and then the data must be rebuilt from
backups and database logs. While a corruption remains undetected, the number of

(FOFSJD
4BGF8SJUF
IBT
CFFO
DSFBUFE
XJUI
3%#.4T
JO
NJOE
"QQMJDBUJPOT
JOUFOEFE
GPS
VTF
XJUI
(FOFSJD
4BGF8SJUF
JODMVEF

CVU
BSF
OPU

MJNJUFE
UP
.JDSPTPGU
&YDIBOHF

.JDSPTPGU
42-
4FSWFS

%#6%#
BOE
0SBDMF
(FOFSJD
4BGF8SJUF
JT
OPU
JOUFOEFE
GPS
VTF
XJUI
BQQMJDBUJPOT

XIFSF
UPSO
QBHFT
BSF
OPU
B
DPODFSO

TVDI
BT
GJMFTIBSFT
PS
'51
TFSWFST

1




8-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

database logs required for recovery increases. This causes the data recovery process to
be more complex and time-consuming.



8.1.2 Data corruption between host and conventional storage
Although data appears to the host to travel directly to the Symmetrix array, it passes
through multiple hardware and software layers. These can lead to problems such as
corruption introduced by errors in the operating system or the I/O driver. Hardware can
also introduce corruption, such as errors in the host adapter, cable and connector
problems, static electricity, and RF noise and interference.
This means that valid data within the RDBMS might arrive corrupted at the storage
device. The storage device writes the data as is because it has no way of validating the
data.

8.1.3 Benefits of checking within Symmetrix
With this feature, the Symmetrix system can perform error checks on data pages handled
within a checksum extent as they are written to the disk. The check occurs before the
write command is acknowledged. If an error is detected within the blocks of the extent,
the I/O can be rejected and/or reported in a phone home connection or logged in the
Symmetrix error log facilities. The error action can be specified by the administrator.
This checking feature minimizes the possibility of data corruption occurring between the
host and the Symmetrix array. It improves the recovery time by flagging the error at the
time of the “write.” When this error condition is raised, and reject I/O is selected, Oracle
takes an action, such as taking the tablespace offline.

8.2 Implementing EMC Double Checksum for Oracle
The symchksum command allows you to perform control operations that manage the
checksum I/O extents.
For example, to enable Oracle checksum checking on the extents of all the devices that
define the current database instance and then to phone home on error, enter:
symchksum enable -type Oracle -phone_home

This command requires an Oracle PAK license.
The following are current restrictions for this feature:
♦ Oracle databases version 8.0 and above are supported.
♦ Data-block size is limited to 32 KB.
♦ Checksum data objects can only be Oracle datafiles, control files, and redo logs.
The Oracle instance being checked must be started with the following init.ora
configuration parameter set to true: db_block_checksum=true

Implementing EMC Double Checksum for Oracle

8-3

Data Protection

8.2.1 Other checksum operations
The following additional checks are available:
♦ MagicNumber – Verify the magic number that appears in Oracle data blocks.
(Enabled by default.)
♦ NonzeroDba – Check for nonzero data block address. The dba stored in a data block
is never zero. (Enabled by default.)
♦ Check_All_Blocks – Apply checks to each block in a write.
♦ Straddle – Check that the write does not straddle known Oracle areas.
♦ Check_dba – Check that the logical address embedded by Oracle is compatible with
the storage address of the writes.
With the addition of these new tests, the output when you list the extents will look
similar to the following:
Symmetrix ID: 000187900671
D E V I C E S

W I T H

C H E C K S U M

E X T E N T S
Action

Checks

R P C
C
A N
D
e h h M h S l Z F i
j o k a k t l
r s
L e n s g D r B D a c
Num Blk
o c e u i B a l B c r
Device Name
Dev Exts Siz Type
g t H m c A d k A t d
------------------------------------------------------------------------------/dev/sdi
/dev/sdj
/dev/sdk
/dev/sdl

047
048
049
04A

16
16
16
15

32b
32b
32b
32b

Oracle
Oracle
Oracle
Oracle

X
X
X
X

.
.
.
.

X
X
X
X

X
X
X
X

X
X
X
X

.
.
.
.

.
.
.
.

.
.
.
.

X
X
X
X

.
.
.
.

.
.
.
.

Use this output to determine which features are enabled on the devices with checksum
extents.
To turn off any of the automatic checksum features, use the -suppress_feature
option and supply the name of the feature, for example:
symchksum -type Oracle enable -suppress_feature MagicNumber

The -suppress_feature option is only for the operations run by default. To turn off
an option that was manually enabled, such as -phone_home, disable the checksum
operation and begin again with a new symchksum enable command.

8.2.2 Enabling checksum options
When using the symchksum enable command, the user can decide to reject the I/O, or
have the Symmetrix phone home when a checksum error is detected.

8-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

If an I/O is not a multiple of the object block size, the user can choose to reject the I/O.
This is called a fractured I/O, and is selected with the -fractured_reject_io option.
When using this option, the -reject_io option must also be used.
When extents are enabled with the -discard option, EMC Double Checksum writes
blocks to disk until a failed block is detected. The -discard option divides a large I/O
into smaller units of 32 KB each. When a checksum failure is detected, all blocks in that
unit and subsequent units are discarded. When using this option, the -reject_io
option must also be used.
The symchksum enable command understands the Oracle database structure. The
feature can be enabled for tablespaces, control files, redo logs, or the entire database.
For Oracle9i and above, if the block size for a tablespace is altered, then the user must
disable and then reenable the extents of the tablespace to ensure that the block size of the
enabled extents match the block size of the tablespace.
Note: When FF or power down occurs, extents are lost. Run the symchksum enable command
again.

8.2.3 Verifying checksum is enabled
The symchksum command also allows you to verify that the datafile’s extents are
currently enabled for checksum checking. This provides an easy way to determine if the
specified tablespace or instance is fully protected by the Symmetrix checksum feature.
The verify action will report if all, some, or none of the Oracle datafile’s extents are
enabled for checksum checking. This is useful in environments where the database
configuration changes frequently. An example of this is if a new datafile is added, but
checksum is not enabled for the new file.
The symchksum verify command understands the Oracle database structure. The
feature can be enabled for tablespaces, control files, redo logs, or the entire database.

8.2.4 Validating for checksum operations
The symchksum command also allows you to validate your Oracle tablespace or
instance for checksum operations without performing any active actions. This is helpful
when you want to know if your database environment is configured to support
Symmetrix checksum functionality without actually making any changes.
If the validate is successful, you can enable EMC Double Checksum on the specified
Oracle database or tablespace. The following items are validated:
♦ Oracle version 8 or higher is installed.
♦ Oracle’s checksum initialization parameter is set (db_block_checksum).
♦ If the Oracle datafile is created on a striped LVM, that the LVM stripe width is a
multiple of the Oracle block size.
♦ Oracle datafile’s block size is less than or equal to 32 KB.

Implementing EMC Double Checksum for Oracle

8-5

Data Protection

♦ The Symmetrix Enginuity version supports the checksum functionality.
♦ Each Symmetrix device has the checksum flag set.
♦ Each Symmetrix device has a supportable number of extents defined.
The symchksum validate command understands the Oracle database structure. The
feature can be enabled for tablespaces, control files, redo logs, or the entire database.

8.2.5 Disabling checksum
The symchksum disable command understands the Oracle database structure. The
feature can be enabled for tablespaces, control files, redo logs, or the entire database.
The symchksum disable command also is used on a device basis. This capability is
not normally used, but is provided in the event the tablespace was dropped before EMC
Double Checksum was disabled for that object.
When the disable action is specified for a Symmetrix device, the -force flag is
required. Disabling extents in this way can cause a mapped tablespace or database to be
only partially protected, therefore, use this option with caution. All the extents
monitored for checksum errors on the specified Symmetrix device will be disabled.

8.3 Implementing Generic SafeWrite for generic applications
Generic SafeWrite (GSW) is used to help protect critical applications from incurring an
incomplete write, and subsequent torn page, due to a failure with a component connected
to the Symmetrix Front End Channel Adapter.

8.3.1 Torn pages: Using Generic SafeWrite to protect applications
A Relational Database Management System (RDBMS), such as Oracle and Microsoft
Exchange, structure data within database files using pages (also referred to as blocks).
Pages within a database are the smallest allocation unit size possible for a database
object (such as a table or a row).
For example, the page size for Microsoft Exchange is 4 KB and for Oracle, though it is
configurable, it is usually set to 8 KB. If an incomplete page is written to a database file,
a corruption to the database will occur. The resulting corruption is commonly referred to
as a torn page.
Torn pages are only detected by most RDBMSs after the corruption is written, when that
area of the database is read, which could be after when the corruption was introduced. In
general, the only recovery from a torn page is to perform a restore from a backup (some
RDBMSs allow page-level restores, while others require a complete database restore).
Torn pages can occur due to failures in various components that lie between the RDBMS
and the storage array. Some of these components include the operating system, file
system, logical volume manager, I/O driver, host bus adapter, Fibre or SCSI link and
storage adapter.

8-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

The EMC Double Checkum Generic SafeWrite feature protects critical applications
from incurring incomplete writes, and subsequent torn pages, due to a failure with a
component connected to the Symmetrix Front End Channel Adapter.
Most often, Generic SafeWriteis used to protect against corruption that occurs when the
HBA and link fails (including server crashes). In this scenario, Generic SafeWrite will
protect against fractured writesoccuring before the data reaches the Symmetrix array.

8.3.2 Why generic?
Generic SafeWrite is deemed generic because the checks performed to ensure complete
data are application independent. For instance, Generic SafeWrite will not perform any
Oracle- or Exchange-specific checksums to verify data integrity. It is important to note
that for Oracle, EMC Double Checksum for Oracle provides a rich set of checks which
can be natively performed by the Symmetrix array. For more information on EMC
Double Checksum for Oracle, consult Section 8.2.

8.3.3 Where to enable Generic SafeWrite
Generic SafeWrite only needs to be enabled for specific devices on the Symmetrix array.
For a RDBMS, Generic SafeWrite only needs to be enabled for devices that support
datafiles. The list below gives an example of database files where the supporting devices
for these files should have Generic SafeWrite enabled:
Microsoft Exchange
♦ .edb files
♦ .stm files
Microsoft SQL Server
♦ .mdf files
♦ .ndf files
Oracle
♦ Data files
♦ Control files
It is recommended to enable Generic SafeWrite for database file devices, though it is
unnecessary to enable it for database log devices. In general, a RDBMS will write to its
respective log file with a 512 byte sector alignment. The RDBMS can therefore
determine the last sector that was correctly written and subsequently discard or rollback
any incomplete transactions.
Note: It is always a best practice to separate the location of database files and log files for a given
database onto unique devices. There are cases, however, where the datafile and log file may share
the same device. In this case, it is still possible to have GSW enabled; however, there will be a
performance impact to the log writes that may impact application performance.

Implementing Generic SafeWrite for generic applications

8-7

Data Protection

There are no restrictions regarding the size of a device or the number of devices where
GSW can be enabled. All device types are supported including devices replicated using
the SRDF and the TimeFinder family of products. It is also supported to enable GSW on
file systems across all logical volume managers as well as on raw devices, given the OS
platforms are supported by the Solutions Enabler Storage Resource Management (SRM)
component. When using file systems on Windows and Linux hosts, for performance
reasons, it is strongly recommended to ensure the file systems are properly aligned with
the storage. For more information regarding filesystem alignment, consult Using diskpar
and diskpart to Align Partitions on Windows Basic and Dynamic Disks available on
Powerlink.

8.3.4 Configuring Generic SafeWrite
To use Generic SafeWrite, you must:
♦ Enable the RDB_cksum device flag on all devices targeted for Generic SafeWrite
use
♦ Run a discover operation to update the physical device information in the SYMAPI
database
♦ Enable Generic SafeWrite on all devices targeted for Generic SafeWrite use via the
symchksum command
8.3.4.1 Setting the RDB_chksum Symmetrix device flag

Before using Generic SafeWrite, the RDB_cksum Symmetrix device flag must be
enabled on all devices targeted for Generic SafeWrite use. This change does not turn
Generic SafeWrite on, it only allows it to be enabled on the specified devices.
The RDB_cksum device flag is set by using the SYMCLI symconfigure command,
which will perform a Symmetrix configuration change. Chapter 1 contains more
information on using symconfigure.
Note: If symconfigure cannot be used, the appropriate device flag is set on the array by a
EMC Customer Support Engineer.

The following is an example command:
symconfigure –sid 54 –f c:\enable_cksum.txt commit

where the c:\enable_cksum.txt file contains the following command:
set device 0015:0019 attribute=RDB_Cksum;

Note: If meta volumes are used, this flag needs to be set for both metaheads and
metamembers.

8-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

8.3.4.2 Enabling Generic SafeWrite

Once the device flags are set on the Symmetrix array, it is possible to use the
symchksum command to enable Generic SafeWrite. Before running the symchksum
command, confirm the following:
♦ The devices enabled for Generic SafeWrite are visible to the host from where the
symchksum command will be run.
♦ Run a symcfg discover command after presenting devices to a host to update
the SYMAPI configuration database with the correct physical drive information.
♦ Using the symchksum command, Generic SafeWrite is enabled by specifying a
specific device, a range of devices, or a device group.
Enabling for a device
To enable Generic SafeWrite for a device, use the command syntax shown in the
example below:
symchksum enable –type generic dev 005 –sid 54
Note: If this is a metadevice, only the metahead needs to be specified.

Enabling for a range of devices
To enable Generic SafeWrite for a contiguous range of devices:
symchksum enable –type generic –range 005:025 –sid 54

Enabling for a device group
To enable Generic SafeWrite for a device group:
symchksum enable –type generic –g sql_data –sid 54
Note: Enabling Generic SafeWrite on a Composite Group (CG) is currently not supported.

The symchksum enable –type generic command automatically sets the Log,
Phone Home, and Generic Double Checksum options as described below:
♦ Log – Indicates that errors will be sent to the Symmetrix error log. These events
should be visible via the symevent command.
♦ Phone Home – Indicates that an error will initiate a call by the Symmetrix to EMC
Customer Service.
♦ Generic – The Generic option allows for two functions to be performed by the
Symmetrix array. First, when an incomplete write is detected, it will be rejected
and the Symmetrix will force the I/O to be retried from the host. Then, if the host is
unavailable to retry the I/O, the write will be discarded, preventing it from being
written to disk.

Implementing Generic SafeWrite for generic applications

8-9

Data Protection

8.3.5 How to disable Generic SafeWrite

(eneric SafeWrite can be disabled using the symchksum disable –type generic
command as shown in the same examples below.
8.3.5.1 Disabling for a device

To disable Generic SafeWrite for a device, use the command syntax shown in the
example below:
symchksum disable –type generic dev 005 –sid 54
Note: If this is a metadevice, only the meta head needs to be specified.
8.3.5.2 Disabling for a range of devices

To disable Generic SafeWrite for a contiguous range of devices:
symchksum disable –type generic –range 005:025 –sid 54
8.3.5.3 Disabling for a device group

To disable Generic SafeWrite for a device group:
symchksum disable –type generic –g sql_data –sid 54

8.3.6 Listing Generic SafeWrite devices
To list which devices are Generic SafeWrite enabled, use the symchksum list
command. Only Generic SafeWrite-enabled devices that are visible to the host running
the symchksum list command are returned.
The following example shows the expected output from the list command, with Generic
for the type and the Log and PhoneH (short for Phone Home) options set as well.

8-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

Figure 8-1

Synchronous replication internals

The symchksum show command is used to look at a specific device. For example:
symchksum show dev 103 -type generic -mb -sid 54

8.3.7 Performance considerations
Performance testing was done with Microsoft Exchange, Microsoft SQL Server and
Oracle on standard devices, and in the case of Microsoft Exchange, also on SRDF/S and
SRDF/A devices. For the Microsoft SQL Server and Oracle performance tests, a TPCC
workload was used. For Microsoft Exchange, the Jetstress performance tool was used.
The results of these tests showed no performance degradation from an application
perspective. The reason application performance remain unaffected is database log
writes, as well as database reads, will be performed normally. When considering log
write response times and database read response times as the main determinates of
application performance with respect to storage, it is expected that client and application
response times will not be greatly affected.
Outside of application performance, there may be a slight increase in the write response
time to the database file devices depending on application profile and usage. In general,
this response time increase should not impact application performance. Writes to a
database file are done asynchronously, therefore write response times to this file are less
of a concern than to the log device.However, there is always a possibility in certain
environments a delay in these asynchronous writes may impact performance.

Implementing Generic SafeWrite for generic applications

8-11

Data Protection

8.4 Syntax and examples
This section contains the symchksum argument descriptions and several examples of
using the SYMCLI symchksum command. Consult the EMC Solutions Enabler
Symmetrix CLI Command Reference for the complete list of syntax and options.
Table 8-1

Background processes for managing a Data Guard environment
Command

Argument

Description

symchksum

list

Lists all devices that currently have checksum checking
enabled.

show

Displays the extents of a specified device that are having
checksum checking performed.

enable

Enables checksum checking on the extents of the specified
devices.

disable

Disables checksum checking on the extents of the specified
devices.

validate

Validates that a specified database or tablespace is able to
have checksum checking enabled.

verify

Verifies whether the specified database or tablespace has
checksum checking enabled on all their devices.

To list the devices on Symmetrix array 3890 with extents being checked for checksum
errors, enter:
symchksum list –sid 3890

To show all the extents of Symmetrix device 0A1 on Symmetrix array 3890 being
checked for checksum errors, enter:
symchksum show dev 0A1 –sid 3890

To enable Checksum on the extents of all the devices that define the current database
instance and then to phone home on error, enter:
symchksum enable –type Oracle –phone_home

To enable Checksum on the extents of all the devices that define the tablespace and then
to log on error, enter:
symchksum enable –type Oracle –tbs SYSTEM

To verify that Oracle tablespace USER01 has Checksum enabled on all the devices that
have defined it, enter:
symchksum verify –type Oracle –tbs USER01

To disable Checksum on the current database instance, enter:
symchksum disable –type Oracle

8-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Data Protection

Note: Disable by device should only be used under special circumstances. For example, this
option can be used to remove extents if a database or a tablespace has been dropped without first
doing a normal disable. In this case, disable by device can be used to remove the extents.

To disable (with force) Checksum for all checksum extents on Symmetrix device 0A1
on Symmetrix unit 3890, enter:
symchksum disable dev 0A1 –sid 3890 -force

Syntax and examples

8-13

Data Protection

8-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Appendix A Related Documents

This appendix presents the following topic:
A.1

Related documents ........................................................................................................ A-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

A-1

Related Documents

A.1 Related documents
The following is a list of related documents that provide more detailed information on
topics described in this Solutions Guide. Many of these documents are on the EMC
Powerlink site (http://powerlink.EMC.com). For Oracle information, consult the Oracle
websites including the main site (http://www.oracle.com), the Oracle Technology
Network (OTN), and Oracle Metalink.
SYMCLI
♦ Solutions Enabler Release Notes (by release)
♦ Solutions Enabler Support Matrix (by release)
♦ Solutions Enabler Symmetrix Device Masking CLI Product Guide (by release)
♦ Solutions Enabler Symmetrix Base Management CLI Product Guide (by release)
♦ Solutions Enabler Symmetrix CLI Command Reference (by release)
♦ Solutions Enabler Symmetrix Configuration Change CLI Product Guide (by
release)
♦ Solutions Enabler Symmetrix SRM CLI Product Guide (by release)
♦ Solutions Enabler Symmetrix Double Checksum CLI Product Guide (by release)
♦ Solutions Enabler Installation Guide (by release)
TimeFinder
♦ Solutions Enabler Symmetrix TimeFinder Family CLI Product Guide (by release)
SRDF
♦ Solutions Enabler Symmetrix SRDF Family CLI Product Guide (by release)
♦ Symmetrix Remote Data Facility (SRDF) Product Guide
♦ Symmetrix Automated Replication UNIX and Windows
Replication Manager
♦ Replication Manager Product Guide
♦ Replication Manager Support Matrix

A-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Related Documents

Oracle
♦ Oracle Data Guard Concepts and Administration
♦ Oracle Database Administrator’s Guide
♦ Oracle Database Backup and Recovery Basics
♦ Oracle Database Backup and Recovery Advanced Users Guide
♦ Oracle Database Performance Tuning Guide
♦ Oracle Database Reference

Related documents

A-3

Related Documents

A-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Appendix B Sample SYMCLI
Group Creation
Commands

This appendix presents the following topic:
B.1

Sample SYMCLI group creation commands .................................................................B-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

B-1

B.1 Sample SYMCLI group creation commands
The following shows how Symmetrix device groups and composite groups are created
for the TimeFinder family of products including TimeFinder/Mirror, TimeFinder/Clone,
and TimeFinder/Snap.
This example shows how to build and populate a device group and a composite group
for TimeFinder/Mirror usage:
Device group:
1. Create the device group:
symdg create dbgroup –type regular
2. Add the standard devices to the group. The database containers reside on five
Symmetrix devices. The device numbers for these are 0CF, 0F9, 0FA, 0FB, and
101:
symld –g device_group add dev 0CF
symld –g device_group add dev 0F9
symld –g device_group add dev 0FA
symld –g device_group add dev 0FB
symld –g device_group add dev 101
3. Associate the BCV devices to the group. The number of BCV devices should be the
same as the number of standard devicesand the same size. The device serial
numbers of the BCVs used in the example are 00C, 00D, 063, 064, and 065.
symbcv –g device_group associate dev 00C
symbcv –g device_group associate dev 00D
symbcv –g device_group associate dev 063
symbcv –g device_group associate dev 064
symbcv –g device_group associate dev 065
Composite group:
1. Create the composite group:
symcg create device_group –type regular
2. Add the standard devices to the composite group. The database containers reside on
five Symmetrix devices on two different Symmetrix arrays. The device numbers for
these are 0CF, 0F9 on the Symmetrix system with the last three digits of 123, and
device numbers 0FA, 0FB, and 101 on the Symmetrix system with the last three
digits of 456:
symcg –g device_group add dev 0CF –sid 123
symcg –g device_group add dev 0F9 –sid 123
symcg –g device_group add dev 0FA –sid 456
symcg –g device_group add dev 0FB –sid 456
symcg –g device_group add dev 101 –sid 456
3. Associate the BCV devices to the composite group. The number of BCV devices
should be the same as the number of standard devicesand the same size. The device
serial numbers of the BCVs used in the example are 00C, 00D, 063, 064, and 065.

B-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

symbcv
symbcv
symbcv
symbcv
symbcv

–cg
–cg
–cg
–cg
–cg

device_group
device_group
device_group
device_group
device_group

associate
associate
associate
associate
associate

dev
dev
dev
dev
dev

00C
00D
063
064
065

–sid
–sid
–sid
–sid
–sid

123
123
456
456
456

This example shows how to build and populate a device group and a composite group
for TimeFinder/Clone usage:
Device group:
1. Create the device group device_group:
symdg create device_group –type regular
2. Add the standard devices to the group. The database containers reside on five
Symmetrix devices. The device numbers for these are 0CF, 0F9, 0FA, 0FB, and
101:
symld –g device_group add dev 0CF
symld –g device_group add dev 0F9
symld –g device_group add dev 0FA
symld –g device_group add dev 0FB
symld –g device_group add dev 101
3. Add the target clone devices to the group. The targets for the clones can be standard
devices or BCV devices. In this example, BCV devices are used. The number of
BCV devices should be the same as the number of standard devices, and the same
size or larger than the paired standard device. The device serial numbers of the
BCVs used in the example are 00C, 00D, 063, 064, and 065.
symbcv –g device_group associate dev 00C
symbcv –g device_group associate dev 00D
symbcv –g device_group associate dev 063
symbcv –g device_group associate dev 064
symbcv –g device_group associate dev 065
Composite group:
1. Create the composite group device_group:
symcg create device_group –type regular
2. Add the standard devices to the group. The database containers reside on five
Symmetrix devices on two different Symmetrix arrays. The device numbers for
these are 0CF, 0F9 on the Symmetrix system with the last three digits of 123, and
device numbers 0FA, 0FB, and 101 on the Symmetrix system with the last three
digits of 456:
symcg –g device_group add dev 0CF –sid 123
symcg –g device_group add dev 0F9 –sid 123
symcg –g device_group add dev 0FA –sid 456
symcg –g device_group add dev 0FB –sid 456
symcg –g device_group add dev 101 –sid 456
3. Add the target for the clones to the device group. In this example, BCV devices are
added to the composite group to simplify the later symclone commands. The
number of BCV devices should be the same as the number of standard devices and

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

B-3

the same size. The device serial numbers of the BCVs used in the example are 00C,
00D, 063, 064, and 065.
symbcv –cg device_group associate dev 00C –sid 123
symbcv –cg device_group associate dev 00D –sid 123
symbcv –cg device_group associate dev 063 –sid 456
symbcv –cg device_group associate dev 064 –sid 456
symbcv –cg device_group associate dev 065 –sid 456
The following example shows how to build and populate a device group and a composite
group for TimeFinder/Snap usage.
Device group:
1. Create the device group device_group:
symdg create device_group –type regular
2. Add the standard devices to the group. The database containers reside on five
Symmetrix devices. The device numbers for these are 0CF, 0F9, 0FA, 0FB, and
101:
symld –g device_group add dev 0CF
symld –g device_group add dev 0F9
symld –g device_group add dev 0FA
symld –g device_group add dev 0FB
symld –g device_group add dev 101
3. Add the VDEVs to the group. The number of VDEVs should be the same as the
number of standard devices and the same size. The device serial numbers of the
VDEVs used in the example are 291, 292, 394, 395, and 396.
symld –g device_group add dev 291 -vdev
symld –g device_group add dev 292 -vdev
symld –g device_group add dev 394 -vdev
symld –g device_group add dev 395 -vdev
symld –g device_group add dev 396 –vdev
Composite group:
1. Create the composite group device_group:
symcg create device_group –type regular
2. Add the standard devices to the composite group. The database containers reside on
five Symmetrix devices on two different Symmetrix arrays. The device numbers for
these are 0CF, 0F9 on the Symmetrix system with the last three digits of 123, and
device numbers 0FA, 0FB, and 101 on the Symmetrix system with the last three
digits of 456:
symcg –g device_group add dev 0CF –sid 123
symcg –g device_group add dev 0F9 –sid 123
symcg –g device_group add dev 0FA –sid 456
symcg –g device_group add dev 0FB –sid 456
symcg –g device_group add dev 101 –sid 456

B-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

3. Add the VDEVs to the composite group. The number of VDEVs should be the
same as the number of standard devices and the same size. The device serial
numbers of the VDEVs used in the example are 291, 292, 394, 395, and 396:
symld –cg device_group add dev 291 –sid 123 -vdev
symld –cg device_group add dev 292 –sid 123 -vdev
symld –cg device_group add dev 394 –sid 456 -vdev
symld –cg device_group add dev 395 –sid 456 -vdev
symld –cg device_group add dev 396 –sid 456 -vdev

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

B-5

B-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Appendix C Related Host
Operations

This appendix presents these topics:
C.1

Overview ........................................................................................................................C-2

C.2

Presenting database copies to a different host................................................................C-4

C.3

Presenting database copies to the same host ................................................................C-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-1

C.1 Overview
Previous sections demonstrated methods of creating a database copy using storage-based
replication techniques. While in some cases, customers create one or more storagebased database copies of the database as “gold” copies (copies that are left in a pristine
state on the array), in most cases they want to present copied devices to a host for
backups, reporting, and other business continuity processes. Mounting storagereplicated copies of the database requires additional array-based, SAN-based (if
applicable), and host-based steps including LUN presentation and masking, host device
recognition, and importing of the logical groupings of devices so that the operating
system and logical volume manager recognize the data on the devices. Copies of the
database can be presented to a new host or presented back to the same host that sees the
source database. The following sections describe the host-specific considerations for
these processes.
Whether using SRDF, TimeFinder, or Replication Manager to create a copy of the
database, there are six essential requirements for presenting the replicated devices and
making the copies available to a host. They include:
♦ Verifying that the devices are presented to the appropriate front-end directors in the
BIN file.
♦ Verifying zoning and LUN presentation through the SAN are configured (if
needed).
♦ Editing configuration information to allow the devices to be seen on the host.
♦ Scanning for the devices on the SCSI paths.
♦ Creating special files (UNIX) or assigning drive letters (Windows).
♦ Making the devices ready for use.
The following sections briefly discuss these steps at a high level.

C.1.1 BIN file configuration
For r the data to be presented to a host, the Symmetrix BIN file must be configured.
LUNs need to be assigned to the hypervolumes, and then presented to front-end director
ports. This configuration can be done by the EMC Customer Engineer (CE) through the
BIN file change request process, or by the customer using software utilities such as
Symmetrix Configuration Manager or EMC ControlCenter.

C.1.2 SAN considerations
Hosts can be attached to a Symmetrix DMX either by direct connectivity (FC-AL,
iSCSI, ESCON, or FICON), or through a SAN using Fibre Channel (FC-SW). When
using direct-connect, all LUNs presented to a front-end port are presented to the host. In
the case of a SAN, additional steps must be considered. These include zoning, which is
a means of enabling security on the switch, and LUN masking, which is used to restrict
hosts to see only the devices that they are meant to see. Also, there are HBA-specific
SAN issues that must be configured on the hosts.

C-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

SAN zoning is a means of restricting FC devices (for example, HBAs and Symmetrix
front-end FC director ports) from accessing all other devices on the fabric. It prevents
FC devices from accessing unauthorized or unwanted LUNs. In essence, it establishes
relationships between HBAs and FC ports using World Wide Names (WWNs). WWNs
are unique hardware identifiers for FC devices. In most configurations, a one-to-one
relationship (the zone) is established between an HBA and FC port, restricting other
HBAs (or FC ports) from accessing the LUNs presented down the port. This simplifies
configuration of shared SAN access and provides protection against other hosts gaining
shared access to the LUNs.
In addition to zoning, LUN masking, which on the Symmetrix system is called Volume
Logix™, can also be used to restrict hosts to see only specified devices down a shared
FC director port. SANs are designed to increase connectivity to storage arrays such as
the Symmetrix. Without Volume Logix, all LUNs presented down a FC port would be
available to all hosts that are zoned to the front-end port, potentially compromising both
data integrity and security.
The combination of zoning and Volume Logix, when configured correctly for a
customer’s environment, ensures that each host only sees the LUNs designated for it.
They ensure data integrity and security, and also simplify the management of the SAN
environment. There are many tools to configure zoning and LUN presentation; primary
among them is EMC SAN Manager™. Identifying specific configuration steps for
zoning and LUN presentation are beyond the scope of this document.
There are several host- and HBA-specific SAN configuration considerations. When
presenting volumes through a SAN, the HBA(s) must be configured correctly for the
fabric. One step in the process is updating the HBA firmware and driver levels to those
validated in the EMC Support Matrix; this is dependent on the type of host and HBA
implemented. Parameter files for particular HBAs may also require modification.
Additional configuration requirements depend on the type of host used. Examples of
additional host configuration needs include updating of the sd.conf file on Solaris
systems, configuration of persistent binding on Solaris hosts, and installation of the
EMC ODM Support Package for Symmetrix on IBM systems.
For additional information on host- and HBA-specific configuration issues, consult the
EMC Host Connectivity guides for each operating system, as well as the HBA-specific
installation and configuration guides.

C.1.3 Final configuration considerations for enabling LUN presentation to hosts
Once the BIN file is configured correctly and connectivity via direct-connect or the SAN
is established, the final steps needed to present the storage are to probe the SCSI bus for
the storage devices, create special files on UNIX systems or assign drive letters on
Windows hosts, and to make the devices ready for device I/O. These host-specific steps
are described by operating system in the following sections.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-3

C.2 Presenting database copies to a different host
There are host-specific requirements to mount volumes when TimeFinder or Replication
Manager are used to create a database copy that is to be presented to a different host.
The following sections describe the processes needed to mount these copies for each of
the major host types. Each section describes the operating-system steps for the
following:
♦ Gathering configuration information prior to the change
♦ Probing the SCSI bus for new devices
♦ Creating special device files (UNIX) or assigning drive letters (Windows)
♦ Importing volume/disk groups (UNIX)
♦ Activating logical volumes (UNIX)
♦ Mounting file systems (if applicable)

C.2.1 AIX considerations
When presenting copies of devices from an AIX environment to a different host from the
one the production copy is running on, the first step is to scan the SCSI bus, which
allows AIX to recognize the new devices. The following demonstrates the steps needed
for the host to discover and verify the disks, bring the new devices under PowerPath
control if necessary, import the volume groups, and mount the file systems (if
applicable).
1. Before presenting the new devices, it is useful to run the following commands and
save the information to compare to after the devices are presented:
lsdev –Cc disk
lspv
syminq

2. Another important step to complete before presenting the devices to the new host is
to understand which volume groups are associated with which physical devices on
the source host. The following commands list the volume groups on the host,
identify for each Oracle volume group which physical devices (hdisks) are used, and
show the relationship between hdisks. Run these commands on the host prior to
making any device changes. This is a precaution only and is to document the
environment should it later need to be restored manually.
lspv

(List all the physical volume identifiers)

lsvg

(List all volume groups, identify Oracle volume groups)

lsvg –p vol_grp

(Run for each Oracle volume, identify hdisks)

syminq

(Find Symmetrix volume numbers for each

Oracle hdisk)

C-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

3. Once the Symmetrix volumes on the source host are identified, determine the
relationship between the source volumes and the target volumes. Finding this
relationship depends on the software used. The following commands are used to
determine the relationships when TimeFinder/Mirror is used as the replication
method:
symmir –g device_group query

4. The next step is for the target host to recognize the new devices. The following
command scans the SCSI buses and examines all adapters and devices presented to
the target system:
cfgmgr –v

Alternatively, the EMC command emc_cfgmgr (found in the
/usr/lpp/EMC/Symmetrix/bin directory) may be executed to probe the SCSI buses
for new devices after the Symmetrix ODM drivers have been installed.
5. Confirm presentation of the new devices by running the following commands:
lsdev –Cc disks
lspv
syminq

6. If EMC PowerPath is installed, use the following commands to place the new
devices under PowerPath control and verify success:
powermt config
powermt display
powermt display dev=all

Once the devices are discovered by AIX, the next step is to import the volume
groups. The key is to keep track of the PVIDs on the source system. The PVID is
the physical volume identifier that uniquely identifies a volume across multiple AIX
systems. When the volume is first included in a volume group, the PVID is assigned
based on the host serial number and the timestamp. In this way, no two volumes
should ever get the same PVID. However, array-based replicating technologies
copy everything on the disk including the PVID.
7. On the production host, use the lspv command to list the physical volumes Locate
the PVID of any disk in the volume group being replicated. On the secondary host,
do an lspv as well. Locate the hdisk that corresponds to the PVID noted in the first
step. Suppose the disk has the designation hdisk33. The volume group can now be
imported using the command:
importvg –y vol_grp hdisk33

8. The volume group descriptor area (VGDA) on every disk in the volume group has a
list of the PVIDs of all the other disks in the volume group. During the import, the
LVM tracks down all the disks using the information in the VGDA. To ensure that
the volume group imported successfully, use the following command to list the
volume groups:
lsvg

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-5

9. If PowerPath is in use on the target host, the import of the volume group must be
executed using a powerdisk rather than an hdisk. The procedure otherwise is the
same. When performing an lspv on the target host, locate the appropriate
hdiskpower## associated with PVID previously obtained. Then, issue the importvg
command:
importvg –y vol_grp hdiskpower##

Once imported, the volume groups are automatically activated by AIX. The next
step is to mount the file systems. If the UID and GIDs are not the same between the
two hosts, run the chown command to change the ownerships of the logical volumes
to the dba user and group that administer the server:
chown dbaadmin:dbagroup /dev/rlvname (Character special device file)
chown dbaadmin:dbagroup /dev/lvname (Block special device file)

10. The first time this procedure is performed, create mount points for the file systems if
raw volumes are not used. The mount points should be made the same as the mount
points for the production file systems.
C.2.1.1

AIX and BCV considerations

TimeFinder/Mirror uses BCVs, which are by default in the “defined” state to AIX. To
change these volumes to the “available” state, execute the following command:
/usr/lpp/EMC/Symmetrix/bin/mkbcv –a

If the devices need to be placed in the “defined” state to AIX, use the following
command:
/usr/lpp/EMC/Symmetrix/bin/rmbcv –a

C.2.2 HP-UX considerations
When presenting clone devices in an HP-UX environment to a host different from the
one the production copy is running on, initial planning and documentation of the source
host environment is first required. The following demonstrates the steps needed for the
target host to discover and verify the disks, bring the new devices under PowerPath
control if necessary, import the volume groups and mount the file systems (if
applicable).
1. Before presenting the new devices, it is useful to run the following commands on the
target host and save the information to compare to output taken after the devices are
presented:
vgdisplay –v | grep “Name”
syminq

(List all volume groups)

(Find Symmetrix volume for each c#t#d#)

2. Another important step to complete before presenting the devices to the new host is
to understand the association of volume groups and physical devices

C-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

(/dev/rdsk/c#t#d#) on the source host since all disks in a volume group must be
replicated together. Additionally, the relationship between the physical devices and
the Symmetrix logical devices (hypervolumes) must be identified. The following
commands list the volume groups on the host, identify for each Oracle volume group
which physical devices are used, and show the relationship between physical devices
and the Symmetrix devices.
vgdisplay –v | grep “VG Name”

(List all volume groups)

vgdisplay –v <vol_grp> | grep “PV Name”

(List PVs for a volume group)
syminq

(Find Symmetrix volume for each c#t#d#)

3. Create map files for each volume group to replicate. The Volume Group Reserve
Area (VGRA) on disk contains descriptor information about all physical and logical
volumes that make up a volume group. This information is used when a volume
group is imported to another host. However, logical volume names are not stored on
disk. When a volume group is imported, the host assigns a default logical volume
name. To ensure that the logical volume names are imported correctly, a map file
generated on the source is created for each volume group and used on the target host
when the group is imported.
vgexport –v –p –m /tmp/vol_grp.map vol_grp

4. Identify the Symmetrix device groups on the source host. The following commands,
when run on the source host, list the device groups created and shows their details:
symdg list (Lists all device groups on the host)
symdg show device_group (Shows specifics of a device group)

5. Once the Symmetrix volumes and applicable device groups on the source host are
identified, identify the relationship between the source volumes and the target
volumes. Finding this relationship depends on the replication software used. The
following commands are used to determine the relationships when
TimeFinder/Mirror is the replication method used:
symmir –g device_group query (TimeFinder/Mirror)

6. After identifying the Symmetrix volumes used as targets, ensure that the target host
recognizes these new devices. The following command scans the SCSI buses and
examines all adapters and devices presented to the target system:
ioscan -fn

7. Create device special files for the volumes presented to the host:
insf -e

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-7

8. Confirm presentation of the new devices by running the following commands and
comparing them to outputs found in step 1:
symcfg discover
syminq

9. If EMC PowerPath is installed, use the following commands to place the new
devices under PowerPath control and verify success:
powermt config
powermt display
powermt display dev=all

(Configures the devices under PowerPath)
(Displays the number of devices per path)
(Displays all of the device detail info)

10. Once the devices are discovered by HP-UX, they need to be identified with their
associated volume groups from the source host to be imported successfully. When
using the vgimport command, specify all of the devices for the volume group to be
imported. Since the target and LUN designations for the target devices are different
from the source volumes, the exact devices must be identified using the syminq and
symmir outputs. Source volume group devices are associated with Symmetrix
source devices through a syminq output. Then Symmetrix device pairings from the
source to target hosts are found from the symmir device group outputs. Finally,
Symmetrix target volume to target host device pairings are made through the
syminq output from the target host.
11. After identifying each of the new /dev/rdsk/c#t#d# devices and their associated
volume groups, create the volume group structures needed to successfully import the
volume groups onto the new host. A directory and group file for each volume group
must be created before the volume group can be imported. Ensure that each volume
group has a unique minor number.
ls –l /dev/*/group

(Identify used minor numbers)

mkdir /dev/vol_grp
mknod /dev/vol_grp/group c 64 0xminor#0000

(minor# must be unique)
12. Import the volume groups onto the target host. Volume group information from the
source host is stored in the Volume Group Reserve Area (VGRA) on each volume
presented to the target host. Volume groups are imported by specifying a volume
group name, if the volume group names are notg used on the target.
vgimport –v –m vg_map_file vol_grp /dev/rdsk/c#t#d#
[/dev/rdsk/c#t#d#]

where vg_map_file is the volume group map file created in step 3, vol_grp is the
volume group name being imported, and c#t#d# are the devices in the specified
volume group.
13. After importing the volume group, activate the volume group.
vgchange –a y vol_grp

C-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

14. Once the volume groups are activated, mount on the target any file systems from the
source host. These file systems may require a file system check using fsck as well.
Add an entry to /etc/fstab for each file system.

C.2.3 Linux considerations
Enterprise releases of Linux from Red Hat and SuSE provide a logical volume manager
for grouping and managing storage. However, it is not common to use the logical
volume manager on Linux. The technique deployed to present and use a copy of Oracle
database on a different host depends on whether or not the logical volume manager is
used on the production host. To access the copy of the database on a secondary host,
follow these steps:
1. Create a mapping of the devices that contain the database to file systems. This
mapping information is used on the secondary host. The mapping can be performed
by using the information in the /etc/fstab file and/or the output from the df
command.
In addition, if the production host does not use logical volume manager, the output
from syminq and symmir/symclone/symsnap command is required to associate the
operating-system device names (/dev/sd<x>) with Symmetrix device numbers on the
secondary host.
2. Unlike other UNIX operating systems, Linux does not have a utility to rescan the
SCSI bus. Any of the following methods allow a user to discover changes to the
storage environment:
•

Rebooting the Linux host

•

Unloading and reload the Fibre Channel or SCSI driver module

•

Making changes to the /proc pseudo-file system to initiate a scan

Although rebooting the Linux host is a viable option, for most enterprise
environment this is unacceptable. The unloading and reload of the Fibre Channel or
SCSI driver module is possible only if the driver is not being used, making it a
highly unreliable technique.
The Linux operating system presents all resources under management by means of a
pseudo file system called /proc. This file system is a representation of in-memory
data structures that are used by the operating system to manage hardware and
software resources. The /proc file system is used to convey resource changes to the
operating system. To initiate a scan of the SCSI bus, execute the following
command on the secondary host:
echo “scsi scan-new-devices” > /proc/scsi/scsi

The devices representing the copy of the database should be available for use on the
secondary host.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-9

3. Confirm presentation of the new devices by running the following commands on the
secondary host:
symcfg discover
syminq

4. If PowerPath is available on the secondary host, use the following command to place
the new devices under the control of PowerPath:
powermt config

Verify the status of the devices by executing the following commands:
powermt display
powermt display dev=all

5. If logical volume manager is used on the production host, import the volume group
definitions back on the secondary host. To do this, use the pvscan, vgimport, and
vgchange commands as follows:
pvscan –novolumegroup
vgimport volume_group_name
vgchange –a y volume_group_name

The pvscan command displays all the devices that are initialized, but not belonging
to a volume group. The command should display all members of the volume groups
that constitute the copy of the database. The vgimport command imports the new
devices and creates the appropriate LVM structures needed to access the data. If
LVM is not used, this step can be skipped.
6. Once the volume groups, if any, are activated, mount on the target any file systems
from the source host. If logical volume manager is not being used, execute syminq
on the secondary host. The output documents the relationship between the operating
system device names (/dev/sd<x>) and the Symmetrix device numbers associated
with the copy of the database. The output from step 1 can be then used to determine
the devices and the file systems that need to be mounted on the secondary host.
These file systems may require a file system check (using fsck) before they can be
mounted. If it does not exist, make an entry to /etc/fstab for each file system.

C.2.4 Solaris considerations
When presenting replicated devices in a Solaris environment to a different host from the
one production is running on, the first step is to scan the SCSI bus which allows the
secondary Solaris system to recognize the new devices. The following steps cause the
host to discover and verify the disks, bring the new devices under PowerPath control if
necessary, import the disk groups, start the logical volumes, and mount the file systems
(if applicable). The following commands assume that VERITAS Volume Manager
(VxVM) is used for logical volume management.

C-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

1. Before presenting the new devices, run the following commands and save the
information to compare to, after the devices are presented:
vxdisk list
vxprint –ht
syminq

2. Another important step to complete before presenting the devices to the new host is
to understand which disk groups are associated with which physical devices on the
source host. The following commands list the disk groups on the host, identify for
each Oracle disk group which physical devices are used, and show the relationship
between hdisks should be run on the host prior to making any device changes. This
is a precaution only and is to document the environment should it need to be restored
manually later.
vxdg list

(List all the disk groups)

vxdisk list

(List all the disks and associated groups)

syminq

(Find Symmetrix volume numbers for each Oracle disk)

3. Once the Symmetrix volumes on the source host are identified, determine the
relationship between the source volumes and the target volumes. Finding this
relationship depends on the software used. The following commands are used to
determine the relationships when TimeFinder or SRDF are the replication methods
used:
symmir –g device_group query
symrdf –g device_group query

(TimeFinder)
(SRDF)

4. The next step is for the target host to recognize the new devices. The following
command scans the SCSI buses, examines all adapters and devices presented to the
target system, and builds the information into the /dev directory for all LUNs found:
drvconfig;devlinks;disks

5. Confirm presentation of the new devices by running the following commands:
format
syminq

6. VERITAS needs to discover the new devices after the OS can see them. To make
VERITAS discover new devices, enter:
vxdctl enable

7. If EMC PowerPath is installed, use the following commands to place the new
devices under PowerPath control and verify success:
powermt config
powermt display
powermt display dev=all

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-11

8. Once VERITAS has found the devices, import the disk groups. The disk group
name is stored in the private area of the disk. To import the disk group, enter:
vxdg –C import diskgroup

Use the –C flag to override the host ownership flag on the disk. The ownership flag
on the disk indicates the disk group is online to another host. When this ownership
bit is not set, the vxdctl enable command actually performs the import when it finds
the new disks.
9. Run the following command to verify that the disk group imported correctly:
vxdg list

10. Activate the logical volumes for the disk groups:
vxvol –g diskgroup startall

11. For every logical volume in the volume group, run fsck must to fix any incomplete
file system unit of work:
fsck -F vxfs /dev/vx/dsk/diskgroup/lvolname

12. Mount the file systems. If the UID and GIDs are not the same between the two
hosts, run the chown command to change the ownerships of the logical volumes to
the DBA user and group that administers the server:
chown dbaadmin:dbagroup /dev/vx/dsk/diskgroup/lvolname
chown dbaadmin:dbagroup /dev/vx/rdsk/diskgroup/lvolname

13. The first time this procedure is performed, create mount points for the file systems,
if raw volumes are not used. The mount points should be made the same as the
mount points for the production file systems.

C.2.5 Windows considerations
To facilitate the management of volumes, especially those of a transient nature such as
BCVs, EMC provides the Symmetrix Integration Utility (SIU). SIU provides the
necessary functions to scan for, register, mount, and unmount BCV devices.
Within the Windows Server environment, logical units (LUNs) are displayed as
PHYSICALDRIVE devices. Use the sympd list command to see the currently
accessible devices on the BCV host, as shown in the following example:
Symmetrix ID: 000185500028
Device Name

Directors

Device

--------------------------- ------------- ------------------------------------Cap
Physical
Sym SA :P DA :IT Config
Attribute
Sts
(MB)
--------------------------- ------------- ------------------------------------\\.\PHYSICALDRIVE4
0000 04B:0 01A:C5 2-Way Mir
N/Grp’d VCM WD
11
\\.\PHYSICALDRIVE5
0002 04B:0 02B:C0 RDF1+Mir
Grp’d
RW
8632
\\.\PHYSICALDRIVE6
000E 04B:0 15A:D1 RDF1+Mir
Grp’d
(M) RW
34526
\\.\PHYSICALDRIVE7
004F 04B:0 01A:D3 2-Way Mir
N/Grp’d
RW
8632

C-12

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

\\.\PHYSICALDRIVE8
\\.\PHYSICALDRIVE9
\\.\PHYSICALDRIVE10
\\.\PHYSICALDRIVE11

00AF
00B3
00B7
00BB

04B:0
04B:0
04B:0
04B:0

16B:C4
01A:D4
16B:C5
01A:D5

2-Way
2-Way
2-Way
2-Way

Mir
Mir
Mir
Mir

Grp’d
Grp’d
Grp’d
Grp’d

(M)
(M)
(M)
(M)

RW
RW
RW
RW

34526
34526
34526
34526

Additionally, view the mapping of these physical drive devices to volume mount points
using the Windows Disk Management console, as shown next.

Figure C-1

Windows Disk Management console

The “Disk x” value represents the similarly numbered PHYSICALDRIVE device from
the sympd command. Thus, Disk 7 is the same device as PHYSICALDRIVE7.
As new devices such as BCVs are presented to a Windows server, the SIU can be used
to scan for and register these devices. The rescan function of the symntctl utility will
rediscover disk devices including newly visible BCV devices:
symntctl rescan

It is possible to use the SIU to manage the mount operations of BCV volumes by
specifying the Symmetrix volume identifier with a mount operation:
symntctl mount –drive W: -sid 028 –symdev 055 –part 1

In the previous example, the -part 1 option specifies the partition on the LUN that is to
be mounted, and is only required if multiple partitions exist on the device. As such, SIU
will mount the volume with Symmetrix volume ID of 055 on Symmetrix 028 to the
drive letter W.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-13

Conversely, it is possible to unmount volumes using SIU. This is recommended prior to
reestablishing the BCV to its source STD volume. In the case of the unmount, only the
drive letter location is required to be presented.
symntctl unmount –drive W:

This command will unmount the volume from the drive letter and dismiss the Windows
cache that relates to the volume. If any running application maintains an open handle to
the volume. SIU will fail and report an error. The administrator should ensure that no
applications are using any data from the required volume; proceeding with an unmount
while processes have open handles is not recommended.
The SIU can identify those processes that maintain open handles to the specified drive,
using the following command:
symntctl openhandle –drive W:

Using this command, it is possible to identify running processes to shut down or
terminate to facilitate the unmount command.

C.2.6 Windows Dynamic Disks
In general, avoid using the Dynamic Disk functionality provided by the Windows Server
environment. This limited Dynamic Disk functionality does not provide necessary API
calls to adequately manage import and deport operations for the disk groups. Refer to
the release notes for SIU to identify how to resolve this situation.
However, while all functionality may be unavailable, it is possible to implement some
limited functionality when using base Dynamic Disk configurations. SIU will allow for
mount and unmount operations against Dynamic Disk configurations, though it will be
unable to import/deport the disk groups.
To import the disk group, use Windows Disk Management. The newly presented disk
group will appear as a “Foreign” group, and may then be imported using the Disk
Management interface. For specific implementation details and limitations, consult the
Windows Help documentation provided by Microsoft.

C.3 Presenting database copies to the same host
A copy of a database in most cases is used on a different host from the one that owns the
source database. The secondary host can then perform operations on the data
independent from the primary host and without any conflicts or issues. In some
circumstances, it is preferred to use the copy of the database on the same host that owns
the source database, or to use two copies of the database on a secondary host. In these
situations, care must be taken as OS environments can experience issues when they own
disks that have identical signatures/descriptor areas. Each operating system and volume
manager has its own unique way of dealing with these issues. The following sections
describe how to manage duplicate disks in a single OS environment.

C-14

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C.3.1 AIX considerations
When presenting database copies back to the same host in an AIX environment, one
must deal with the fact that the OS now sees the source disk and an identical copy of the
source disk. This is because the replication process copies not only the data part of the
disk, but also the system part, which is known as the Volume Group Descriptor Area
(VGDA). The VGDA contains the physical volume identifier (PVID) of the disk, which
must be unique on a given AIX system.
The issue with duplicate PVIDs prevents a successful import of the copied volume group
and has the potential to corrupt the source volume group. Fortunately, AIX provides a
way to circumvent this limitation. AIX 4.3.3 SP8 and later provides the recreatevg
command to rebuild the volume group from a supplied set of hdisks or powerdisks.
Use syminq to determine the hdisks or powerdisks that belong to the volume group
copy. Then, issue either of the two commands:
recreatevg –y replicavg_name -l lvrename.cfg hdisk## hdisk##
hdisk ## …
recreatevg –y replicavg_name -l lvrename.cfg hdiskpower##
hdiskpower## hdiskpower## …

where the ## represents the disk numbers of the disks in the volume group. The
recreatevg command gives each volume in the set of volumes a new PVID, and also
imports and activates the volume group.
The lvrename.cfg file can be used to assign new alternative names to the logical
volumes. If the file is not provided, AIX provides default LV names.
A successful recreatevg command varies on the volume group and performs JFS log
replay if necessary for any cooked file systems. Default mount points for each file
system are created in /etc/file systems using the format /xx/oldmountpoint where xx is
specified with the –L parameter. If desired, mount points can be changed by editing
/etc/file systems. The command mount –a can be used to mount the file systems of the
replicated database.

C.3.2 HP-UX considerations
Presenting database copies in an HP-UX environment to the same host as the production
copy is nearly identical to the process used for presenting the copy to a different host.
The primary differences are the need to use a different name for the volume groups and
the need to change the volume group IDs on the disks.
1. Before presenting the new devices, it is useful to run the following commands on the
target host and save the information to compare to outputs taken after the devices are
presented:
vgdisplay –v | grep “Name”
syminq

(List all volume groups)
(Find Symmetrix volume for each c#t#d#)

2. Another important step to complete before presenting the devices to the new host is
to understand the association of volume groups and physical devices

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-15

(/dev/rdsk/c#t#d#) on the source host since all disks in a volume group must be
replicated together. Additionally, the relationship between the physical devices and
the Symmetrix logical devices (hypervolumes) must be identified. The following
commands list the volume groups on the host, identify for each Oracle volume group
which physical devices are used, and show the relationship between physical devices
and the Symmetrix devices:
vgdisplay –v | grep “VG Name” (List all volume groups)
vgdisplay –v <vol_grp> | grep “PV Name”

(List PVs for a volume group)
syminq

(Find Symmetrix volume for each c#t#d#)

3. Create map files for each volume group to be replicated. The Volume Group
Reserve Area (VGRA) on disk contains descriptor information about all physical
and logical volumes that make up a volume group. This information is used when a
volume group is imported to another host. However, logical volume names are not
stored on disk. When a volume group is imported, the host assigns a default logical
volume name. To ensure that the logical volume names are imported correctly, a
map file generated on the source is created for each volume group and used on the
target host when the group is imported.
vgexport –v –p –m /tmp/vol_grp.map vol_grp

4. Identify the Symmetrix device groups on the source host. The following commands,
when run on the source host, list the device groupsn created and shows their details:
symdg list

play

symdg show device_group

(Shows specifics of a device group)

5. Once the Symmetrix volumes and applicable device groups on the source host are
identified, identify the relationship between the source volumes and the target
volumes. Finding this relationship depends on the replication software used. The
following commands are used to determine the relationships when
TimeFinder/Mirror is the replication method used:
symmir –g device_group query (TimeFinder)

6. After identifying the Symmetrix volumes used as targets, ensure that the target host
recognizes these new devices. The following command scans the SCSI buses and
examines all adapters and devices presented to the target system:
ioscan -fn

7. Create device special files for the volumes presented to the host:
insf -e

C-16

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

8. Confirm presentation of the new devices is confirmed by running the following
commands and comparing them to output created in step 1:
symcfg discover
syminq

9. If EMC PowerPath is installed, use the following commands to place the new
devices under PowerPath control and verify success:
powermt config
powermt display
powermt display dev=all

(Configures the devices under PowerPath)
(Displays the number of devices per path)
(Displays all of the device detail info)

10. Once the devices are found by HP-UX, identify them with their associated volume
groups from the source host so that they can be imported successfully. When using
the vgimport command, specify all of the devices for the volume group to be
imported. Since the target and LUN designations for the target devices are different
from the source volumes, the exact devices must be identified using the syminq and
symmir output. Source volume group devices can be associated with Symmetrix
source devices through syminq output. Then Symmetrix device pairings from the
source to target hosts are found from the symmir device group output. And finally,
Symmetrix target volume to target host device pairings are made through the
syminq output from the target host.
11. Change the volume group identifiers (VGIDs) on each set of devices making up
each volume group. For each volume group, change the VGID on each device using
the following:
vgchgid /dev/rdsk/c#t#d# [/dev/rdsk/c#t#d#] . . .

12. After changing the VGIDs for the devices in each volume group, create the volume
group structures needed to successfully import the volume groups onto the new host.
A directory and group file for each volume group must be created before the volume
group is imported. Ensure each volume group has a unique minor number and is
given a new name.
ls –l /dev/*/group

(Identify used minor numbers)

mkdir /dev/newvol_grp
mknod /dev/newvol_grp/group c 64 0xminor#0000

(minor# must be unique)
13. Import the volume groups onto the target host. Volume group information from the
source host is stored in the VGRA on each volume presented to the target host.
Volume groups are imported by specifying a volume group name, if the volume
group names are not used on the target.
vgimport –v –m vg_map_file vol_grp /dev/rdsk/c#t#d#
[/dev/rdsk/c#t#d#]

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-17

where vg_map_file is the volume group map file created in step 3, vol_grp is the
volume group name being imported, and c#t#d# is the device in the specified
volume group.
14. After importing the volume groups, activate them:
vgchange –a y vol_grp

15. Once the volume groups are activated, mount on the target any file systems from the
source host. These file systems may require a file system check using fsck as well.
An entry should be made to /etc/fstab for each file system.

C.3.3 Linux considerations
Presenting database copies back to the same Linux host is possible only if the production
volumes are not under the control of the logical volume manager. Linux logical volume
manager does not have utility such as vgchgid to modify the UUID (universally unique
identifier) written in the private area of the disk.
For a Oracle database not under LVM management, the procedure to import and access
a copy of the production data on the same host is similar to the process for presenting the
copy to a different host. The following steps are required:
1. Execute syminq and symmir/symclone/symsnap to determine the relationship
between the Linux device name (/dev/sd<x>), the Symmetrix device numbers that
contain the production data, and the Symmetrix device numbers that hold the copy
of the production data. In addition, note the mount points for the production devices
as listed in /etc/fstab and the output from the command df.
2. Initiate the scan of SCSI bus by running the following command as root:
echo “scsi scan-new-devices” > /proc/scsi/scsi

3. If PowerPath is in use on the production host, bring the new devices under
PowerPath control:
powermt config

Verify the status of the devices:
powermt display
powermt display dev=all

4. Use the syminq or sympd list commands to display, in addition to the production
devices, the devices that contain the database copy. Note the Linux device name
associated with the newly added devices.
5. Using the mapping information collected in step 1, mount the file systems on the
new devices to the appropriate mount points. Note that the copy of the data has to be
mounted at a different location since the production database volumes are still
mounted and accessible on the production host. For ease of management, it is
recommended to create a directory structure similar to the production devices.

C-18

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

For example, assume that the production database consists of one Symmetrix device,
000 accessed by the operating system by device name /dev/sdb1 and mounted at /u01.
Furthermore, assume that the copy of the database is available on Symmetrix device 100
and device name is /dev/sdc1. It is recommended to unmount /dev/sdc1 at mount point
/copy/u01.
The total number of hard disks presented on a Linux host varies anywhere from 128 to
256 depending on the version of the Linux kernel. When presenting copies of database
back to the production host, ensure that the total devices does not exceed the limit.

C.3.4 Solaris considerations
Presenting database copies to a Solaris host using VERITAS volume manager where the
host can see the individual volumes from the source volume group is not supported other
than with Replication Manager. Replication Manager provides “production host” mount
capability for VERITAS.
The problem is that the VERITAS Private Area on both the source and target volumes is
identical. A vxdctl enable finds both volumes and gets confused as to which are the
source and target.
To get around this problem, the copied volume needs to be processed with a vxdisk init
command. This re-creates the private area. Then, a vxmake using a map file from the
source volume created with a vxprint –hvmpsQq –g dggroup can be used to rebuild the
volume group structure after all the c#t#d# numbers are changed from the source disks
to the target disks. This process is risky and difficult to script and maintain and is not
recommended by EMC.

C.3.5 Windows considerations
The only difference for Windows when bringing back copies of volumes to the same
Windows server is that duplicate volumes or volumes that appear to be duplicates are not
supported in a cluster configuration.

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

C-19

C-20

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

Appendix D Sample Database
Cloning Scripts

This appendix presents the following topic:
D.1

Sample script to replicate a database............................................................................. D-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

D-1

D.1 Sample script to replicate a database
The following example shows a Korn shell script where the requirements are to replicate
the Oracle database onto a different host than the primary database using TimeFinder
functionality. In this case, BCVs on the Symmetrix system are established to the
primary database volumes. After the establish is complete, the tablespace names are
found and each is put into hot backup mode. The BCVs are then split from the
standards. Two device groups, DATA_DG and LOG_DG, are used to split the log
information separately from the rest of the Oracle data.
The main script is called main.ksh, which contains callouts to perform all the additional
required tasks.
#!/bin/ksh
#
############################################################
#
# Main script - Note: This assumes that the device groups
#
DATA_DG and LOG_DG are already created.
#
############################################################
############################################################
# Define Variables
############################################################
ORACLE_SID=oratest
export ORACLE_SID
ORACLE_HOME=/oracle/oracle10g
export ORACLE_HOME
SCR_DIR=/opt/emc/scripts
CLI_DIR=/usr/symcli/bin
DATA_DG=data_dg
LOG_DG=logs_dg
#############################################################
############################################################
# Establish the BCVs for each device group
############################################################
${SCR_DIR}/establish.ksh
RETURN=$?
if [ $RETURN != 0 ]; then
exit 1
fi
############################################################
# Get the tablespace names using sqlplus
############################################################
su - oracle -c ${SCR_DIR}/get_tablespaces_sub.ksh
RETURN=$?
if [ $RETURN != 0 ]; then
exit 2

D-2

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

fi
############################################################
# Put the tablespaces into hot backup mode
############################################################
su - oracle -c ${SCR_DIR}/begin_hot_backup_sub.ksh
############################################################
# Split the DATA_DG device group
############################################################
${SCR_DIR}/split_data.ksh
RETURN=$?
if [ $RETURN != 0 ]; then
exit 3
fi
############################################################
# Take the tablespaces out of hot backup mode
############################################################
su - oracle -c ${SCR_DIR}/end_hot_backup_sub.ksh
############################################################
# Split the LOG_DG device group
############################################################
${SCR_DIR}/split_log.ksh
RETURN=$?
if [ $RETURN != 0 ]; then
exit 4
fi
echo "Script appeared to work successfully"
exit 0
=================================================================
#!/bin/ksh
############################################################
# establish.ksh
#
This script initiates a BCV establish for the $DATA_DG
#
and $LOG_DG device groups on the Production Host.
############################################################
############################################################
# Define Variables
############################################################
CLI_BIN=/usr/symcli/bin
DATA_DG=data_dg
LOG_DG=log_dg
############################################################
Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

D-3

# Establish the DATA_DG and LOG_DG device groups
############################################################
${CLI_BIN}/symmir -g ${DATA_DG} -noprompt establish
RETURN=$?
if [ $RETURN != 0 ]; then
ERROR_DATE=‘date‘
echo "Split failed for Device Group ${DATA_DG}!!!"
echo "Script Terminating."
echo
echo "establish: failed"
echo "$ERROR_DATE: establish: failed to establish ${DATA_DG}"
exit 1
fi
${CLI_BIN}/symmir -g ${LOG_DG} -noprompt establish
RETURN=$?
if [ $RETURN != 0 ]; then
ERROR_DATE=‘date‘
echo "Establish failed for Device Group ${LOG_DG}!!!"
echo "Script Terminating."
echo
echo "establish: failed"
echo "$ERROR_DATE: establish: failed to establish ${LOG_DG}"
exit 2
fi
############################################################
# Cycle ${CLI_BIN}/symmir query for status
############################################################
RETURN=0
while [ $RETURN = 0 ]; do
${CLI_BIN}/symmir -g ${LOG_DG} query | grep SyncInProg \
> /dev/null
RETURN=$?
REMAINING=‘${CLI_BIN}/symmir -g ${LOG_DG} query | grep MB | \
awk ’{print $3}’‘
echo "$REMAINING MBs remain to be established."
echo
sleep 10
done
RETURN=0
while [ $RETURN = 0 ]; do
${CLI_BIN}/symmir -g ${DATA_DG} query | grep SyncInProg \
> /dev/null
RETURN=$?
REMAINING=‘${CLI_BIN}/symmir -g ${DATA_DG} query | grep MB | \
awk ’{print $3}’‘
echo "$REMAINING MBs remain to be established."
echo
sleep 10
done
exit 0

D-4

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

=================================================================
#!/bin/ksh
############################################################
# get_tablespaces_sub.ksh
#
This script queries the Oracle database and returns with
#
a list of tablespaces which is then used to identify
#
which tablespaces need to be placed into hotbackup mode.
############################################################
############################################################
# Define Variables
############################################################
SCR_DIR=/opt/emc/scripts
############################################################
# Get the tablespace name using sqlplus
############################################################
sqlplus internal <<EOF > /dev/null
set echo off;
spool ${SCR_DIR}/tablespaces.tmp;
select tablespace_name from dba_tablespaces;
spool off;
exit
EOF
############################################################
# Remove extraneous text from spool file
############################################################
cat ${SCR_DIR}/tablespaces.tmp | grep -v "TABLESPACE_NAME" | \
grep -v "-" |grep -v "rows selected." \
> ${TF_DIR}/tablespaces.txt
############################################################
# Verify the creation of the tablespace file
############################################################
if [ ! -s ${SCR_DIR}/tablespaces.txt ]; then
exit 1
fi
exit 0
=================================================================
#!/bin/ksh
############################################################
# begin_hot_backup_sub.ksh
#
This script places the oracle database into hot backup
#
mode.
############################################################

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

D-5

############################################################
# Define Variables
############################################################
SCR_DIR=/opt/emc/scripts
############################################################
# Do a log switch
############################################################
sqlplus internal <<EOF
alter system archive log current;
exit
EOF
############################################################
# Put all tablespaces into hot backup mode
############################################################
TABLESPACE_LIST=‘cat ${SCR_DIR}/tablespaces.txt‘
for TABLESPACE in $TABLESPACE_LIST; do
sqlplus internal <<EOF
alter tablespace ${TABLESPACE} begin backup;
exit
EOF
done
exit 0
=================================================================
#!/bin/ksh
############################################################
# split_data.ksh
#
This script initiates a Split for the $DATA_DG Device
#
group on the Production Host.
############################################################
############################################################
# Define Variables
############################################################
CLI_BIN=/usr/symcli/bin
DATA_DG=data_dg
############################################################
# Split the DATA_DG device group
############################################################
${CLI_BIN}/symmir -g ${DATA_DG} -noprompt -instant split
RETURN=$?
if [ $RETURN != 0 ]; then
ERROR_DATE=‘date‘
echo "Split failed for Device Group ${DATA_DG}!!!"
echo "It is not safe to continue..."

D-6

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

echo
echo
echo
echo
exit
fi

"Script Terminating."
"split_data: failed"
"$ERROR_DATE: split_data: failed to split ${DATA_DG}"
1

############################################################
# Cycle ${CLI_BIN}/symmir query for status
############################################################
RETURN=0
while [ $RETURN = 0 ]; do
${CLI_BIN}/symmir -g ${DATA_DG} query | grep SplitInProg \
> /dev/null
RETURN=$?
REMAINING=‘${CLI_BIN}/symmir -g ${DATA_DG} query | grep MB | \
awk ’{print $3}’‘
echo "$REMAINING MBs remain to be split."
echo
sleep 5
done
exit 0
=================================================================
#!/bin/ksh
############################################################
# end_hot_backup_sub.ksh
#
This script ends the hot backup mode for the oracle
#
database. The script is initiated by the end_hot_backup
#
scrips
############################################################
############################################################
# Define Variables
############################################################
SCR_DIR=/opt/emc/scripts
###########################################################
# Take all tablespaces out of hotbackup mode
############################################################
TABLESPACE_LIST=‘cat ${SCR_DIR}/tablespaces.txt‘
for TABLESPACE in $TABLESPACE_LIST; do
sqlplus internal <<EOF
alter tablespace ${TABLESPACE} end backup;
exit
EOF
done
############################################################
# Do a log switch
Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

D-7

############################################################
sqlplus internal <<EOF
alter system archive log current;
exit
EOF
exit 0
=================================================================
#!/bin/ksh
############################################################
# split_log.ksh
#
This script initiates a Split for the $LOG_DG Device
#
group on the Production Host.
############################################################
############################################################
# Define Variables
############################################################
CLI_BIN=/usr/symcli/bin
LOG_DG=log_dg
############################################################
# Split the LOG_DG device group
############################################################
${CLI_BIN}/symmir -g ${LOG_DG} -noprompt -instant split
RETURN=$?
if [ $RETURN != 0 ]; then
ERROR_DATE=‘date‘
echo "Split failed for Device Group ${LOG_DG}!!!"
echo "It is not safe to continue..."
echo "Script Terminating."
echo
echo "split_data: failed"
echo "$ERROR_DATE: split_data: failed to split ${LOG_DG}"
exit 1
fi
############################################################
# Cycle ${CLI_BIN}/symmir query for status
############################################################
RETURN=0
while [ $RETURN = 0 ]; do
${CLI_BIN}/symmir -g ${LOG_DG} query | grep SplitInProg \
> /dev/null
RETURN=$?
REMAINING=‘${CLI_BIN}/symmir -g ${LOG_DG} query | grep MB | \
awk ’{print $3}’‘
echo "$REMAINING MBs remain to be split."
echo
sleep 5

D-8

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

done
exit 0
=================================================================

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide

D-9

D-10

Oracle Databases on EMC Symmetrix Storage Systems 1.1 Solutions Guide