Add a Node to an Existing Oracle 2 Node RAC 11g R2 Cluster on (OEL-5.4)
This
article assumes the following:
1.
All
shared disk storage for the existing Oracle RAC is based on iSCSI using a Network
Storage Server; namely Openfiler Release 2.99.1 (Final) x86_64.
2.
The
existing Oracle RAC does not use shared Oracle homes for the Grid
Infrastructure or Database software.
3.
The
existing Oracle RAC is not using Grid Naming Service (GNS) to assign IP
addresses.
4.
The
existing Oracle RAC database is administrator-managed (not policy-managed).
5. During the creation of the
existing Oracle RAC, the installation of Oracle Grid Infrastructure and the
Oracle Database software were only performed from one node in the RAC cluster —
namely from racnode1 as the grid and oracle user account respectively. The
Oracle Universal Installer (OUI) on that particular node would then use the ssh and scp commands to run remote commands
on and copy the Oracle software to all other nodes within the RAC cluster. The grid and oracle user accounts on the node
running the OUI (runInstaller) had to be trusted by all other
nodes in the cluster. This meant that the grid and oracle user accounts had to run the
secure shell commands (ssh or scp) on the Linux server executing
the OUI (racnode1)
against all other Linux servers in the cluster without being prompted for a
password. The same security requirements hold true when extending Oracle RAC.
Linux
Servers are configured as follows:
Nodes
|
RAC1
|
RAC2
|
RAC3
(New)
|
OPENFILER
(STORAGE)
|
Instance Name
|
Racdb1
|
Racdb2
|
Racdb3
|
|
Database Name
|
Racdb
|
Racdb
|
Racdb
|
|
Operating System
|
OEL 5.4 –(x86_64)
|
OEL 5.4 –(x86_64)
|
OEL 5.4 –(x86_64)
|
Openfiler-2.99.1
–(x86_64)
|
Public IP
|
192.168.7.151
Subnet:
255.255.255.0
Gateway:
192.168.7.254
|
192.168.7.154
Subnet:
255.255.255.0
Gateway:
192.168.7.254
|
192.168.7.156
Subnet: 255.255.255.0
Gateway:
192.168.7.254
|
192.168.7.159
Subnet:
255.255.255.0
|
Private IP
|
192.168.8.151
255.255.255.0
|
192.168.8.154
255.255.255.0
|
192.168.8.156
255.255.255.0
|
|
Virtual IP
|
192.168.7.152
|
192.168.7.155
|
192.168.7.157
|
|
SCAN name
|
racnode-cluster-scan
|
racnode-cluster-scan
|
racnode-cluster-scan
|
|
SCAN IP
|
192.168.7.161
|
192.168.7.161
|
192.168.7.161
|
The following
is a conceptual look at what the environment will look like after adding the
third Oracle RAC node (racnode3) to the
cluster.
Step 1: For OS preparation:
Follow
the step from 1 to 17 (Step by Step process of Two node RAC configuration)
Step 2: Configure Access to the Shared Storage
From the Openfiler Storage Control Center home page, log in as an administrator. The default administration login credentials for Openfiler are:https://192.168.7.159:446/
Username: openfiler
Password: password
Navigate
to [System] / [Network Setup]. The "Network Access
Configuration" section (at the bottom of the page) allows an administrator
to setup networks and/or hosts that will be allowed to access resources
exported by the Openfiler appliance. Add the new Oracle RAC node individually
rather than allowing the entire network to have access to Openfiler resources.
Before
the iSCSI client on the new Oracle RAC node can access the shared volumes, it
needs to be granted the appropriate permissions to the associated iSCSI
targets. From the Openfiler Storage Control Center, navigate to [Volumes]
/ [iSCSI Targets]. Under the "Target Configuration" sub-tab,
use the pull-down menu to select one of the current RAC iSCSI targets in the
section "Select iSCSI Target" and then click the [Change]
button.
Click on
the grey sub-tab named "Network ACL" (next to "LUN Mapping"
sub-tab). For the currently selected iSCSI target, change the
"Access" for the new Oracle RAC node from 'Deny' to 'Allow'
and click the [Update] button. This needs to be performed for all of the
RAC iSCSI targets.
Configure the iSCSI (initiator) service
Next, start the iscsid service and enable it to automatically start when the system boots. Also configure the iscsi service to automatically start which logs into iSCSI targets needed at system startup.
[root@racnode3 ~]# service iscsid start
Turning off network shutdown. Starting iSCSI daemon: [ OK ]
[ OK ]
[root@racnode3 ~]# chkconfig iscsid on
[root@racnode3 ~]# chkconfig iscsi on
|
[root@racnode3 ~]# iscsiadm -m discovery -t sendtargets -p openfiler1-priv
192.168.7.159:3260,1 iqn.2006-01.com.openfiler:racdb.fra1
192.168.7.159:3260,1 iqn.2006-01.com.openfiler:racdb.crs1
192.168.7.159:3260,1 iqn.2006-01.com.openfiler:racdb.data1
|
Log In to iSCSI Targets
Manually log in to each of the available iSCSI targets using the iscsiadm command-line interface.
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs1 -p 192.168.7.159 -l
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.data1 -p 192.168.7.159 -l
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.fra1 -p 192.168.7.159 -l
|
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs1 -p 192.168.7.159 --op update -n node.startup -v automatic
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.data1 -p 192.168.7.159 --op update -n node.startup -v automatic
[root@racnode3 ~]# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.fra1 -p 192.168.7.159 --op update -n node.startup -v automatic
|
Create Persistent Local SCSI Device Names
Create persistent local SCSI device names for each of the iSCSI target names using udev. Having a consistent local SCSI device name and which iSCSI target it maps to, helps to differentiate between the three volumes when configuring ASM. Although this is not a strict requirement since we will be using ASMLib 2.0 for all volumes, it provides a means of self-documentation to quickly identify the name and location of each iSCSI volume.Start by creating the following rules file /etc/udev/rules.d/55-openiscsi.rules on the new Oracle RAC node.
# /etc/udev/rules.d/55-openiscsi.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh %b",SYMLINK+="iscsi/%c/part%n"
|
[root@racnode3 ~]# mkdir -p /etc/udev/scripts
|
#!/bin/sh
# FILE: /etc/udev/scripts/iscsidev.sh
BUS=${1}
HOST=${BUS%%:*}
[ -e /sys/class/iscsi_host ] || exit 1
file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname"
target_name=$(cat ${file})
# This is not an open-scsi drive
if [ -z "${target_name}" ]; then
exit 1
fi
# Check if QNAP drive
check_qnap_target_name=${target_name%%:*}
if [ $check_qnap_target_name = "iqn.2004-04.com.qnap" ]; then
target_name=`echo "${target_name%.*}"`
fi
echo "${target_name##*.}"
|
[root@racnode3 ~]# chmod 755 /etc/udev/scripts/iscsidev.sh
|
[root@racnode3 ~]# service iscsi stop
Logging out of session [sid: 1, target: iqn.2006-01.com.openfiler:racdb.crs1, portal: 192.168.7.159,3260]
Logging out of session [sid: 3, target: iqn.2006-01.com.openfiler:racdb.data1, portal: 192.168.7.159,3260]
Logging out of session [sid: 4, target: iqn.2006-01.com.openfiler:racdb.fra1, portal: 192.168.7.159,3260]
Logout of [sid: 1, target: iqn.2006-01.com.openfiler:racdb.crs1, portal: 192.168.7.159,3260]: successful
Logout of [sid: 3, target: iqn.2006-01.com.openfiler:racdb.data1, portal: 192.168.7.159,3260]: successful
Logout of [sid: 4, target: iqn.2006-01.com.openfiler:racdb.fra1, portal: 192.168.7.159,3260]: successful
Stopping iSCSI daemon: [ OK ]
[root@racnode3 ~]# service iscsi start
iscsid dead but pid file exists
Turning off network shutdown. Starting iSCSI daemon: [ OK ]
[ OK ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.fra1, portal: 192.168.7.159,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs1, portal: 192.168.7.159,3260]
Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.data1, portal: 192.168.7.159,3260]
Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.fra1, portal: 192.168.7.159,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs1, portal: 192.168.7.159,3260]: successful
Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.data1, portal: 192.168.7.159,3260]: successful
[ OK ]
|
[root@racnode3 ~]# ls -l /dev/iscsi/*
/dev/iscsi/acfs1:
total 0
lrwxrwxrwx 1 root root 9 Apr 30 10:30 part -> ../../sdb
lrwxrwxrwx 1 root root 10 Apr 30 10:30 part1 -> ../../sdb1
/dev/iscsi/crs1:
total 0
lrwxrwxrwx 1 root root 9 Apr 30 10:30 part -> ../../sdc
lrwxrwxrwx 1 root root 10 Apr 30 10:30 part1 -> ../../sdc1
/dev/iscsi/data1:
total 0
lrwxrwxrwx 1 root root 9 Apr 30 10:30 part -> ../../sde
lrwxrwxrwx 1 root root 10 Apr 30 10:30 part1 -> ../../sde1
/dev/iscsi/fra1:
total 0
lrwxrwxrwx 1 root root 9 Apr 30 10:30 part -> ../../sdd
lrwxrwxrwx 1 root root 10 Apr 30 10:30 part1 -> ../../sdd1
|
Step 3:
To make the volumes available on the
new Oracle RAC node enter the following command.
[root@racnode3 ~]# /usr/sbin/oracleasm scandisks
Reloading
disk partitions: done
Cleaning
any stale ASM disks...
Scanning
system for ASM disks...
Instantiating
disk "CRSVOL1"
Instantiating
disk "FRAVOL1"
Instantiating
disk "DATAVOL1"
|
Verify that the new Oracle RAC node
has identified the disks that are marked as Automatic Storage Management disks.
[root@racnode3 ~]# /usr/sbin/oracleasm listdisks
CRSVOL1
DATAVOL1
FRAVOL1
|
Step 4: Verify New Node (HWOS)
From one of the active nodes in the existing
cluster, log in as the Oracle Grid Infrastructure owner and run cvuqdisk at the post-hardware installation to ensure that racnode3 (the Oracle RAC node to be added) is ready from the
perspective of the hardware and operating system.
[root@racnode1 ~]# su - grid
[grid@racnode1 ~]$ echo $GRID_HOME
/u01/app/11.2.0/grid
[grid@racnode1 ~]$ echo $ORACLE_HOME
/u01/app/11.2.0/grid
[grid@racnode1 ~]$ $GRID_HOME/bin/cluvfy stage -post hwos -n racnode3
|
If the CVU was successful, the command will end with:
Post-check for hardware and operating system setup was successful.
Otherwise, the CVU will print meaningful
error messages
Step 5: Verify Peer (REFNODE)
As the Oracle Grid Infrastructure owner, run
the CVU again, this time to determine the readiness of the new Oracle RAC node.
Use the comp
peer option to obtain a detailed
comparison of the properties of a reference node that is part of your existing
cluster environment with the node to be added in order to determine any
conflicts or compatibility issues with required Linux packages, kernel
settings, and so on. Specify the reference node (racnode1 in this example) against which you want CVU to compare the node to be added (the node(s) specified after the -n option). Also provide the name of the Oracle inventory O/S group as well as the name of the OSDBA O/S group.
[grid@racnode1 ~]$ $GRID_HOME/bin/cluvfy comp peer -refnode racnode1 -n racnode3 -orainv oinstall -osdba dba -verbose
|
Invariably, the CVU will report:
Verification of peer compatibility was unsuccessful.
This is due to the fact that the report
simply looks for mismatches between the properties of the nodes being compared.
Certain properties will undoubtedly differ. For example, the amount of
available memory, the amount of free disk space for the Grid_home, and
the free space in /tmp will rarely match exactly. These such mismatches can
be safely ignored. Differences, however, with kernel settings and required
Linux packages should be addressed before extending the cluster. Step 6: Verify New Node (NEW NODE PRE)
Use CVU as the Oracle Grid Infrastructure owner one last time to determine the integrity of the cluster and whether it is ready for the new Oracle RAC node to be added.
[grid@racnode1 ~]$ $GRID_HOME/bin/cluvfy stage -pre nodeadd -n racnode3 -fixup -verbose
|
If the CVU was successful, the command will end with:
Pre-check for node addition was successful.
Otherwise, the CVU will create fixup scripts
(if the -fixup option was specified) with instructions to fix the
cluster or node if the verification fails. When the shared storage is Oracle ASM and ASMLib is being used, there are cases where you may receive the following error from CVU:
ERROR:
PRVF-5449 : Check of Voting Disk location "ORCL:CRSVOL1(ORCL:CRSVOL1)" failed on the following nodes:
racnode3:No such file or directory
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
As documented in Oracle BUG #10310848, this
error can be safely ignored. The error was a result of having the voting disks
stored in Oracle ASM which is a new feature of Oracle 11g Release 2.
Step 7: Extend Oracle Grid Infrastructure for a Cluster to
the New Node
Set the
environment variable export IGNORE_PREADDNODE_CHECKS=Y
before running addNode.sh in order to
bypass the node addition pre-check; otherwise, the silent node addition will
fail without showing any errors to the console.
If you are not using GNS (like me):
[grid@racnode1 ~]$ id
uid=1100(grid)
gid=1000(oinstall)
groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)
[grid@racnode1
~]$ cd $GRID_HOME/oui/bin
[grid@racnode1
bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={racnode3}"
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={racnode3-vip}"
 |
If you are using GNS:
[grid@racnode1 ~]$ id
uid=1100(grid)
gid=1000(oinstall)
groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)
[grid@racnode1
~]$ cd $GRID_HOME/oui/bin
[grid@racnode1
bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={racnode3}"
|
If the command is successful, you
should see a prompt similar to the following:
The
following configuration scripts need to be executed as the "root"
user in each new cluster node.
Each
script in the list below is followed by a list of nodes.
/u01/app/oraInventory/orainstRoot.sh
#On nodes racnode3
/u01/app/11.2.0/grid/root.sh
#On nodes racnode3
To
execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The
Cluster Node Addition of /u01/app/11.2.0/grid was successful.
Please
check '/tmp/silentInstall.log' for more details.
|
Run the orainstRoot.sh and root.sh
commands on the new Oracle RAC node. The root.sh script performs the work of
configuring Grid Infrastructure on the new node and includes adding High
Availability Services to the /etc/inittab so that CRS starts up when the
machine starts. When root.sh completes, all services for Oracle Grid
Infrastructure will be running.
[root@racnode3 ~]# /u01/app/oraInventory/orainstRoot.sh
[root@racnode3
~]# /u01/app/11.2.0/grid/root.sh
|
It is best practice to run the CVU
from one of the initial nodes in the Oracle RAC one last time to verify the
cluster is integrated and that the new node has been successfully added to the
cluster at the network, shared storage, and clusterware levels.
[grid@racnode1 ~]$ $GRID_HOME/bin/cluvfy stage -post nodeadd -n racnode3 -verbose |
Step
8: Verify Oracle
Grid Infrastructure for a Cluster on the New Node
After extending Oracle Grid Infrastructure,
run the following tests to verify the install and configuration was successful
from the new Oracle RAC node as the grid user. If successful, the Oracle Clusterware daemons, the TNS listener,
the ASM instance, etc. should be started by the root.sh script. Check CRS Status
[grid@racnode3 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
|
Check Cluster Nodes
[grid@racnode3 ~]$ olsnodes -n
racnode1 1
racnode2 2
racnode3 3
|
Oracle TNS Listener Process
[grid@racnode3 ~]$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_SCAN2
LISTENER
|
Confirming Oracle ASM Function for Oracle Clusterware Files
If you installed the OCR and voting disk files on Oracle ASM, then use the following command syntax as the Grid Infrastructure installation owner to confirm that your Oracle ASM installation is running.
[grid@racnode3 ~]$ srvctl status asm -a
ASM is running on racnode3,racnode1,racnode2
ASM is enabled.
|
Check Oracle Cluster Registry (OCR)
[grid@racnode3 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3592
Available space (kbytes) : 258528
ID : 1546531707
Device/File Name : +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
|
Check Voting Disk
[grid@racnode3 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 7fe9ad5212f84fb5bf48192cede68454 (ORCL:CRSVOL1) [CRS]
Located 1 voting disk(s).
|
Step 9: Extend
Oracle Database Software to the New Node
From one of the active nodes in the
existing Oracle RAC, log in as the Oracle owner (oracle) and execute the addNode.sh
script to install and configure the Oracle Database software on the new node.
Like with Oracle Grid Infrastructure, the only mode to run the addNode.sh
script is with the -silent option. The GUI installation method is no longer
available.
[oracle@racnode1 ~]$ id
uid=1101(oracle)
gid=1000(oinstall) groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)
[oracle@racnode1
~]$ cd $ORACLE_HOME/oui/bin
[oracle@racnode1
bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={racnode3}"
|
If the command is successful, you
should see a prompt similar to the following:
...
The
following configuration scripts need to be executed as the "root"
user in each new cluster node.
Each
script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh
#On nodes racnode3
To
execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node
The
Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1 was
successful.
Please
check '/tmp/silentInstall.log' for more details.
|
Run the root.sh command on the new
Oracle RAC node as directed:
[root@racnode3 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh |
Change Group Ownership of 'oracle' Binary when using Job Role Separation
When using ASMLib, you can always
determine the OSASM group using the oracleasm configure command.
[root@racnode3 ~]# /usr/sbin/oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=grid
ORACLEASM_GID=asmadmin
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
|
For other platforms, check installAction<date>.log
for the OSASM setting.
As the grid user, run the setasmgidwrap
command to set the $ORACLE_HOME/bin/oracle binary to the proper group
ownership.
[root@racnode3 ~]# su - grid
[grid@racnode3
~]$ cd $GRID_HOME/bin
[grid@racnode3
bin]$ ./setasmgidwrap o=/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
[grid@racnode3
bin]$ ls -l /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
-rwsr-s--x
1 oracle asmadmin 232399319 Apr 26 18:25
/u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle*
|
This
change is only required for the Oracle Database software ($ORACLE_HOME/bin/oracle). Do
not modify the $GRID_HOME/bin/oracle
binary ownership for Oracle Grid Infrastructure
Step 10: Add New Instance to the Cluster Database
Use either the Oracle Database Configuration Assistant (DBCA) GUI or the SRVCTL command-line interface to add a new instance to the existing cluster database running on the new Oracle RAC node.Database Configuration Assistant — GUI Method
To use the GUI method, log in to one of the active nodes in the existing Oracle RAC as the Oracle owner (oracle) and execute the DBCA.[oracle@racnode1 ~]$ dbca & |
Screen Name
|
Response
|
Welcome Screen
|
Select Oracle Real Application Clusters database.
|
Operations
|
If your database is administrator-managed, select Instance
Management.
If your database is policy-managed, then the Instance
Management option is not available. To increase the number of database
instances, add more nodes to the server pool.
|
Instance Management
|
Select Add an Instance.
|
List of cluster databases
|
From the List of Cluster Databases page, select the active
Oracle RAC database to which you want to add an instance. Enter user name and
password for the database user that has SYSDBA privileges.
|
List of cluster databases instances
|
Review the existing instances for the cluster database and
click Next to add a new instance.
|
Instance naming and node selection
|
On the Adding an Instance page, enter the instance name
in the field at the top of this page if the instance name that DBCA provides
does not match your existing instance naming scheme. Then select the target
node name from the list and click Next.
|
Instance Storage
|
Expand the Tablespaces, Datafiles, and Redo
Log Groups nodes to verify a new UNDO tablespace and Redo Log Groups of a
new thread are being created for the purpose of the new instance then click Finish.
|
Summary
|
Review the information on the Summary dialog and click OK.
|
Progress
|
The DBCA displays a progress dialog showing DBCA
performing the instance addition operation.
|
End of Add Instance
|
When DBCA completes the instance addition operation, DBCA
displays a dialog asking whether you want to perform another operation. Click
No to exit from the DBCA.
|
Step 11: Verify New Instance
[oracle@racnode3 ~]$ srvctl status database -d racdb -v
Instance racdb1 is running on node racnode1 with online services racdbsvc.idevelopment.info. Instance status: Open.
Instance racdb2 is running on node racnode2 with online services racdbsvc.idevelopment.info. Instance status: Open.
Instance racdb3 is running on node racnode3 with online services racdbsvc.idevelopment.info. Instance status: Open.
|
SQL> select inst_id, instance_name, status,
2 to_char(startup_time, 'DD-MON-YYYY HH24:MI:SS') as "START_TIME"
3 from gv$instance order by inst_id;
INST_ID INSTANCE_NAME STATUS START_TIME
---------- ---------------- ------------ --------------------
1 racdb1 OPEN 26-APR-2012 22:12:38
2 racdb2 OPEN 27-APR-2012 00:16:50
3 racdb3 OPEN 26-APR-2012 23:11:22
|
No comments:
Post a Comment