Installation and Configuration of Oracle 10g RAC Database Cluster with Error/Bug Fix

[milind]

In this article I am going to Explain the steps to install and configure Oracle 10g RAC on Linux (Redhat Enterprise Linux 5) using NFS (Network File System).

Index (Click on topic to navigate it Directly)

01. Requirements
02. Installation and Configuration of OS
03. Configure Private Network
04. Configure Hostname
05. User and Key Configuration
06. Setting User Profile
07. Configure Kernel Parameters and User limitation
08. NFS Configuration
09. Oracle Cluster Installation
10. Install Oracle Database Software
11. Create a Database using the DBCA
12. Check the Status of the RAC

1) Requirement

Pre-installation Check list:
1) Servers with two NIC Cards (for Public and Private Network)
2) OS: Redhat Enterprise Linux 64bit or 32 bit.
3) NFS Share
4) Software: Oracle Clusterware, Oracle RAC Software (Can be Downloaded from Oracle Website).

[milind]

We have used RHEL 5 x64 bit OS for Setting up Oracle Cluster. While installation of OS make sure you select following Packages:

• X Window System
• GNOME Desktop Environment
• Editors
• Graphical Internet
• Server Configuration Tools
• FTP Server
• Development Tools
• Legacy Software Development
• Administration Tools
• System Tools

Following are the other Packages which are required for Oracle Cluster setup.

binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
elfutils-libelf-devel-static-0.125
gcc-4.1.2
gcc-c++-4.1.2
glibc-2.5-24
glibc-common-2.5
glibc-devel-2.5
glibc-headers-2.5
kernel-headers-2.6.18
ksh-20060214
libaio-0.3.106
libaio-devel-0.3.106
libgcc-4.1.2
libgomp-4.1.2
libstdc++-4.1.2
libstdc++-devel-4.1.2
libXp-1.0.0-8.1
libXp-devel-1.0.0-8.1
make-3.81
rsh-server-0.17-40
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11

You can verify if all packages are installed on your system or not by using command rpm -qa <package name>

Make sure all the package are installed on all the nodes which are going to setup inside this cluster.

[milind]

To setup Oracle the servers you are setting up should have two ethernet cards one for public network and another for Private Network.

To setup secondary private IP edit /etc/sysconfig/network-scripts/ifcfg-eth1 (or another eth configuration file) and add private ip

NODE 1:

ONBOOT=yes
IPADDR=10.10.10.2
NETMASK=255.255.255.0

NODE 2:

ONBOOT=yes
IPADDR=10.10.10.3
NETMASK=255.255.255.0

Then bring eth1 online using command ifup.

[root@rac01 RPMS]# ifup eth1
[root@rac01 RPMS]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:23:7D:DB:E6:AA
inet addr:115.124.120.10 Bcast:115.124.120.255 Mask:255.255.255.0
inet6 addr: fe80::223:7dff:fedb:e6aa/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:224264 errors:0 dropped:0 overruns:0 frame:0
TX packets:97528 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:34705747 (33.0 MiB) TX bytes:20520900 (19.5 MiB)
Memory:fbe60000-fbe80000

eth1 Link encap:Ethernet HWaddr 00:23:7D:DB:E6:AB
inet addr:10.10.10.2 Bcast:10.10.10.255 Mask:255.255.255.0
inet6 addr: fe80::223:7dff:fedb:e6ab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:0 (0.0 b) TX bytes:6189 (6.0 KiB)
Memory:fbee0000-fbf00000

[milind]

Set the hostname of all the nodes and add following entries in /etc/hosts

#Public IP of the Cluster Nodes
115.124.120.10 rac1.esds.co.in rac1
115.124.120.11 rac2.esds.co.in rac2

#Private IP of Clustered Nodes
10.10.10.2 pvt1.esds.co.in pvt1
10.10.10.3 pvt2.esds.co.in pvt2

#Virtual IP of both the Nodes
115.124.120.14 rac1-vip.esds.co.in rac1-vip
115.124.120.9 rac2-vip.esds.co.in rac2-vip

[milind]

Setup Oracle User and Group

Node 1:

[root@rac01 ~]# groupadd oinstall
[root@rac01 ~]# groupadd dba
[root@rac01 ~]# useradd -g oinstall -G dba oracle
[root@rac01 ~]# passwd oracle
[root@rac01 ~]# id oracle
uid=500(oracle) gid=500(oinstall) groups=500(oinstall),501(dba) context=root:system_r:unconfined_t:SystemLow-SystemHigh
[root@rac01 ~]#

Node 2:

[root@rac2 ~]# /usr/sbin/groupadd -g 500 oinstall
[root@rac2 ~]# /usr/sbin/groupadd -g 501 dba
[root@rac2 ~]# /usr/sbin/useradd -m -u 500 -g oinstall -G dba oracle
[root@rac2 ~]# password oracle

Setup Keys for without password logins

RAC1 (Node1):
Generate RSA Keys:

[oracle@rac1 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
68:46:61:67:5e:35:a7:ce:51:d4:73:95:7c:b5:e9:3d oracle@rac1.esds.co.in
[oracle@rac1 ~]$

Generate DSA Keys:

[oracle@rac1 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
4d:75:9d:64:c9:51:32:07:45:b7:a2:1c:14:11:ad:10 oracle@rac1.esds.co.in

RAC2 (Node2):
Generate RSA Keys:

[oracle@rac2 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
2e:b2:Data Centers:9f:78:c0:85:e9:76:f7:a5:40:3f:0d:ee:00 oracle@rac2.esds.co.in
[oracle@rac2 ~]$

Generate DSA Keys:

[oracle@rac2 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
ac:d7:33:a6:08:e1:c6:34:93:2e:9d:59:96:c0:29:45 oracle@rac2.esds.co.in
[oracle@rac2 ~]$

Setting up Keys on Nodes

RAC1:

[oracle@rac1 ~]$ cat /home/oracle/.ssh/id_dsa.pub >> /home/oracle/.ssh/authorized_keys
[oracle@rac1 ~]$ cat /home/oracle/.ssh/id_rsa.pub >> /home/oracle/.ssh/authorized_keys
[oracle@rac1 ~]$ ssh oracle@rac2 cat /home/oracle/.ssh/id_dsa.pub >> /home/oracle/.ssh/authorized_keys
oracle@rac2's password:
[oracle@rac1 ~]$ ssh oracle@rac2 cat /home/oracle/.ssh/id_rsa.pub >> /home/oracle/.ssh/authorized_keys
oracle@rac2's password:
[oracle@rac1 ~]$

RAC2:

[oracle@rac2 ~]$ cat /home/oracle/.ssh/id_dsa.pub >> /home/oracle/.ssh/authorized_keys
[oracle@rac2 ~]$ cat /home/oracle/.ssh/id_rsa.pub >> /home/oracle/.ssh/authorized_keys
[oracle@rac2 ~]$ ssh oracle@rac1 cat /home/oracle/.ssh/id_dsa.pub >> /home/oracle/.ssh/authorized_keys
[oracle@rac2 ~]$ ssh oracle@rac1 cat /home/oracle/.ssh/id_rsa.pub >> /home/oracle/.ssh/authorized_keys

Verify Keys:

Now make sure that contents of /home/oracle/.ssh/authorized_keys should be same on all nodes

[oracle@rac2 ~]$ cat /home/oracle/.ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBALr3Fwu2Dp17y1WkLsGJGr5gcJz6eI pSG1z3fmdC//3qKvBKMm0IGip4KSPwR/jro0cw9jrCQXuU6QWqaKXrAZdcatN/GTDHM9z6DlvMNGDNbd9U9qBLJ6dZ2FmOAO5kMDyem+Y34gZpFT Z4BZBhP8vsbxwHvqh34RrDJAvBsBgeeVMR4Vbgd5hXmUpP9UtH NvnlhMSfLosuY9LdCl4ScMooqtCJ1UpfNn0Uc2D9ORSz50t1lb wJCMPj+t9+FJ1Ecb9eomAF+/3vcq3/vPWYs5YSdV7G/93l1a/Z9OnjTsnQAAAIBBRbdvdcBJFaZpEwZvy5fVu/g+8NrwbHqWUGtOwPAMPuxhu68ZYPw1Efw7bAdspUOs//UQ4N/472Vhiwqvwhzam4LG9k0kV0uN5mC3x8amBLtHCM4EZwFBJwwBV mkLkegUIxwsG4HZB3ZlfwiSStHQWXpoigiUjI+Nn3NHQ+hbWw= = oracle@rac2.esds.co.in
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwPubSmpXb4HWbo0uMCZ3yZ N0CM0EoNdvjx8pOseJbrJKrmaskc5M9Zd6/ZqXbwwJIrRAFda/kWvkAKe7lsX9YJ9ohkazQXauwB6TtRtTK7fU9+d5vkJbza6p/Y1GDu8NyI4EIv3FTxyB5XE6ueB0JYXJQAsWooPo1aUjVLaEXDC TC0L0Aisj7BFVpgaXxh9ykdjfepD4LtAnvMq5Gm8fLHYAgOlWV frRLnjUEXhK2DlTf/HVY+QKTLIWpFHk+fswp14w== oracle@rac2.esds.co.in
ssh-dss AAAAB3NzaC1kc3MAAACBALlBvcG6/phsD7RnUDpsaoIBxXiBLXt51A4l89JThqqPtTDwW7V5d7+guzs OEiGRuOvStiDTcdVHqDk4tlMIV5lE3AAAAFQDl4h7Q8Z5YTLLN dsm1JyQlbxuhqQAAAIBVMSHwD1pqgHwrHp4J/qRl/uGzp+KB9iPbzR26szPddRm0TzmfiAY/4Q0VNzyyG34QnwuJvV+qBntDEQK0+ow/mXixFEu6Fx38ZXMGa+W+BcAVu2nr+e7KFkQZWzeJNA7acRijmD 2ZjhxKHmcyZCLIcCZ9/+BJout+ouuDBGcvjAAAAIEAo1+J8yo2X2SrmStGSJ9Ug4Y1zov 2l/jiFCRMPedYACD5OhFWk6khwc94FpsKybxL3UsoL85Gjzs/FsUL0M77M1T6Wm5menWJP/kZWNU3vgKy01w51w9cEqXSwkzDU312iCzq5ayoekRsl5uPqfBq 3vIWYpc8q6gk/fpHPiTEplc= oracle@rac1.esds.co.in
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA42BV+IikvTRCR5DTtKido9 weVfk/87NbxmFxjqpwfa+Y5pmWWb532Am+2bUD9HPI8A/JRS+yP2QtGkqDsTMc9pVa2iDM20a98xu5w3yqqXtgrIxPISz5a r/t1RkyAojJ93lZHyrTftsV0NwQH3jjZ3M+zKctkHCLVrhumI7ue 1SncmmR0nMIahu6wuqJs9SMTXR/F0diBcL7KfrfN/eA0PQyUgNY6NZ+6ObcVOhtl3xIzLPsriL5mPJvyUSlYIj09AmQ == oracle@rac1.esds.co.in

[milind]

Add following lines in .bash_profile of oracle user. You can find the .bash_profile file at /home/oracle/.bash_profile location.

# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR

ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

[milind]

Configure Kernel Parameters

Now execute Following command on shell at once (Just copy paste) on all the Clustered nodes

[oracle@rac2 ~]$ cat >> /etc/sysctl.conf << EOF
>kernel.shmall = 2097152
>kernel.shmmax = 536870912
>kernel.shmmni = 4096
>kernel.sem = 250 32000 100 128
>fs.file-max = 658576
>net.ipv4.ip_local_port_range = 1024 65000
>net.core.rmem_default = 262144
>net.core.wmem_default = 262144
>net.core.rmem_max = 1048536
>net.core.wmem_max = 1048536
>EOF

[oracle@rac1 ~]$ /sbin/sysctl -p

Setting Shell Limits for the oracle User

[oracle@rac2 ~]$cat >> /etc/security/limits.conf << EOF
>oracle soft nproc 2047
>oracle hard nproc 16384
>oracle soft nofile 1024
>oracle hard nofile 65536
>EOF

[oracle@rac2 ~]$cat >> /etc/pam.d/login << EOF
session required /lib/security/pam_limits.so
EOF

[oracle@rac2 ~]$cat >> /etc/profile << EOF
>if [ \$USER = "oracle" ]; then
>if [ \$SHELL = "/bin/ksh" ]; then
>ulimit -p 16384
>ulimit -n 65536
>else
>ulimit -u 16384 -n 65536
>fi
>umask 022
>fi
>EOF

[oracle@rac2 ~]$cat >> /etc/csh.login << EOF
>if ( \$USER == "oracle" ) then
>limit maxproc 16384
>limit descriptors 65536
>umask 022
>endif
>EOF

Configure the Hangcheck Timer

[oracle@rac2 ~]$modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
cat >> /etc/rc.d/rc.local << EOF
[oracle@rac2 ~]$modprobe hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
EOF

[milind]

1) Create two directories on both the servers named /u01 and /u02.

mkdir /u01
mkdir /u02

2) Now mount these directories to NFS share. Add following line in /etc/fstab of your Server

nas1:/share1 /u01 nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768 ,wsize=32768,actimeo=0 0 0
nas1:/share2 /u02 nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768 ,wsize=32768,actimeo=0 0 0

Example:

115.124.124.13:/data/rac /u01 nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768 ,wsize=32768,actimeo=0 0 0
115.124.124.13:/data/rac2 /u02 nfs rw,bg,hard,nointr,tcp,vers=3,timeo=300,rsize=32768 ,wsize=32768,actimeo=0 0 0

3) Create the shared CRS Configuration, Voting Disk files and other required directoried on these NFS Shares.

touch /u01/crs_configuration
touch /u01/voting_disk
mkdir -p /u01/crs/oracle/product/10.2.0/crs
mkdir -p /u01/app/oracle/product/10.2.0/db_1
mkdir -p /u01/oradata
chown -R oracle:oinstall /u01 /u02

[milind]

1) Download clusterware and database software in the /u02 directory and extract it.

2) Login as oracle user on both the servers and set User Equivalence using following commands

exec /usr/bin/ssh-agent $SHELL
/usr/bin/ssh-add

3) go to clusterware directory and run oracle installer.

cd /u02/10gR2/clusterware
./runInstaller

BUG FIX:- While installation of oracle 10 clusterware on RHEL 5 it gives error of OS Compatibility. To fix this issue open file /u02/10gR2/clusterware/install/oraparam.ini and search for

[Certified Versions]
Linux=redhat-3,SuSE-9,redhat-4,UnitedLinux-1.0,asianux-1,asianux-2

Add redhat-5 in Certified Linux Version section. Your file now should look like:

[Certified Versions]
Linux=redhat-3,SuSE-9,redhat-4,redhat-5,UnitedLinux-1.0,asianux-1,asianux-2

4) On Welcome Screen press "Next" to Continue

5) Specify Inventory Directory and Credentials: Press Next to continue

6) Specify Home Details: You can set any Name for the installation and installation path. We have set Name as RAC and path as /u01/app/oracle/product/10.2.0/clust

7) Product-Specification Prerequisite Check: Oracle will scan your system and check for Prerequisites. Press Next once its complete.

8) Specify Cluster Configuration: Here you will have to mention the Cluster name and Network Configuration of the Server.

Edit existing cluster node as per your settings and add new nodes.

Make sure that you add all nodes which are going to be in cluster.

9) Specify Network Interface Usage: The system detect network configuration of the cluster. Press next to continue

Continue in Next Post

[milind]

Continue From Above

10) Specify Oracle Cluster Registry: Select External Redundancy and specify location as "/u01/crs_configuration". Press Next

11) Specify Voting Disk Location: Select External Redundancy and specify location as "/u01/voting_disk". Press Next

12) Summary: click the "Install" button to continue.

13) Once Installation gets completed following window will appear asking execute two commands from root user

Execute Following Commands on Node 1 as root user

/u01/app/oracle/oraInventory/orainstRoot.sh
/u01/crs/oracle/product/10.2.0/clust/root.sh

BUG FIX 1:-
When you run /u01/crs/oracle/product/10.2.0/clust/root.sh on Second node you might get following Errors

Running vipca(silent) for configuring nodeapps
/u01/app/oracle/product/crs/jdk/jre//bin/java: error while loading shared libraries:libpthread.so.0: cannot open shared object file: No such file or directory

to fix this issue edit /u01/app/oracle/product/10.2.0/clust/bin/vipca add following (Red colored or on line number 124):

if [ "$arch" = "i686" -o "$arch" = "ia64" ]
then
LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL
fi
unset LD_ASSUME_KERNEL

BUG FIX 2:-

Once above changes made when you run /u01/app/oracle/product/10.2.0/clust/bin/vipca you will get following Error

[root@rac2 bin]# /u01/app/oracle/product/10.2.0/clust/bin/vipca
Error 0(Native: listNetInterfaces:[3])
[Error 0(Native: listNetInterfaces:[3])]

Now you will have to configure Virtual IP's Manually. You can configure it using following commands

[root@rac2 bin]# pwd
/u01/app/oracle/product/10.2.0/clust/bin
[root@rac2 bin]# ./oifcfg setif -global eth0/115.124.120.0:public
[root@rac2 bin]# ./oifcfg setif -global eth1/10.10.10.1:cluster_interconnect
[root@rac2 bin]# ./oifcfg getif
eth0 115.124.120.0 global public
eth1 10.10.10.0 global cluster_interconnect
[root@rac2 bin]#

Now Run VIPCA (On Next Post)