Hello friends,
The Oracle DBA occasionally needs to update some of the Oracle Clusterware network configurations. Today let's check how to update the private network part.
The Environment
Oracle RAC with 2 nodes: ol8-19-rac1 and ol8-19-rac2
Oracle Linux Server release 8.8
Grid Version
34318175;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34318175)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34139601;ACFS RELEASE UPDATE 19.16.0.0.0 (34139601)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)
DB Version
34086870;OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)
When you see <dbenv>, load the DB HOME variables.
When you see <gridenv>, load the GRID HOME variables.
IPs
# Public
192.168.56.161 ol8-19-rac1.localdomain ol8-19-rac1
192.168.56.162 ol8-19-rac2.localdomain ol8-19-rac2
# Virtual
192.168.56.163 ol8-19-rac1-vip.localdomain ol8-19-rac1-vip
192.168.56.164 ol8-19-rac2-vip.localdomain ol8-19-rac2-vip
# Private
10.1.2.161 ol8-19-rac1-priv.localdomain ol8-19-rac1-priv
10.1.2.162 ol8-19-rac2-priv.localdomain ol8-19-rac2-priv
As the grid user, take a backup of profile.xml on all cluster nodes before proceeding.
[[email protected] ~]$ cp -p $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp
[[email protected] ~]$ ll $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp
-rw-r--r--. 1 oracle oinstall 2171 May 27 20:09 /u01/app/19.0.0/grid/gpnp/ol8-19-rac1/profiles/peer/profile.xml.bkp
Now, let’s check the Oracle CRS networks and the Linux networks.
It’s possible to identify that Oracle is using the “eth2 10.1.2.0” network as his private one.
[[email protected] ~]$ oifcfg getif
eth1 192.168.56.0 global public
eth2 10.1.2.0 global cluster_interconnect,asm
[[email protected] ~]$ ip -c -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1 UP 192.168.56.161/24 192.168.56.163/24 192.168.56.166/24 192.168.56.165/24 fe80::a00:27ff:fe4f:b8aa/64
eth2 UP 10.1.2.161/24 169.254.1.211/19 fe80::a00:27ff:fe01:a7de/64
eth3 UP 10.1.3.161/24 fe80::a00:27ff:fef3:172b/64
[[email protected] ~]$ ip -c -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1 UP 192.168.56.162/24 192.168.56.164/24 192.168.56.167/24 fe80::a00:27ff:fe76:b954/64
eth2 UP 10.1.2.162/24 169.254.26.118/19 fe80::a00:27ff:fe7f:8663/64
eth3 UP 10.1.3.162/24 fe80::a00:27ff:fea8:d7c3/64
Let’s also check the ASM Listener configuration and take note.
[[email protected] ~]$ srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.2.0
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
[[email protected] ~]$ srvctl status listener -l ASMNET1LSNR_ASM
Listener ASMNET1LSNR_ASM is enabled
Listener ASMNET1LSNR_ASM is running on node(s): ol8-19-rac2,ol8-19-rac1
[[email protected] ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM
Firstly, let’s check if both nodes are able to reach the network we intend to use.
[[email protected] tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.014 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.018 ms
--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.010/0.014/0.018/0.003 ms
[[email protected] tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.168 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.151 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.171 ms
--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2068ms
rtt min/avg/max/mdev = 0.151/0.163/0.171/0.013 ms
[[email protected] tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.153 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.176 ms
--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2081ms
rtt min/avg/max/mdev = 0.153/0.161/0.176/0.017 ms
[[email protected] tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.025 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.015 ms
--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2037ms
rtt min/avg/max/mdev = 0.010/0.016/0.025/0.007 ms
Since everything seems OK, now we will start changing the private network. We will change it from the subnet eth2:10.1.2.0 to eth3:10.1.3.0. As the grid user, proceed as below in only one of the nodes.
[[email protected] ~]$ oifcfg setif -global eth3/10.1.3.0:cluster_interconnect,asm
[[email protected] ~]$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 10.1.3.0
[[email protected] ~]$ srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force
--The error below is expected, as we are only testing if the previous listener still exists.
[[email protected] ~]$ lsnrctl stop ASMNET1LSNR_ASM
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 28-MAY-2023 07:55:09
Copyright (c) 1991, 2023, Oracle. All rights reserved.
TNS-01101: Could not find listener name or service name ASMNET1LSNR_ASM
Check if the ASM Listener was created correctly.
[[email protected] ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNEWLSNR_ASM
[[email protected] ~]$ srvctl config listener -asmlistener
Name: ASMNEWLSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.3.0
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:
In the 19c version, we also have one more resource to handle, the ASM Network.
[[email protected] ~]$ srvctl config asmnetwork
ASM network 1 exists
Subnet IPv4: 10.1.2.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
[[email protected] ~]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1
As the root user, proceed as below.
[[email protected] scripts]# srvctl remove asmnetwork -netnum 1 -force
[[email protected] scripts]# srvctl add asmnetwork -netnum 1 -subnet 10.1.3.0
Now let’s restart the entire cluster.
[[email protected] ~]# crsctl stop cluster -all -f
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac1'
...
CRS-2673: Attempting to stop 'ora.cssd' on 'ol8-19-rac1'
CRS-2677: Stop of 'ora.cssd' on 'ol8-19-rac1' succeeded
[[email protected] ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[[email protected] ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[[email protected] ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[[email protected] ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
Check if everything is OK.
[[email protected] tmp]$ crsctl status res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE ol8-19-rac1 STABLE
ONLINE ONLINE ol8-19-rac2 STABLE
ora.chad
ONLINE ONLINE ol8-19-rac1 STABLE
ONLINE ONLINE ol8-19-rac2 STABLE
ora.net1.network
ONLINE ONLINE ol8-19-rac1 STABLE
ONLINE ONLINE ol8-19-rac2 STABLE
ora.ons
ONLINE ONLINE ol8-19-rac1 STABLE
ONLINE ONLINE ol8-19-rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE ol8-19-rac1 STABLE
OFFLINE OFFLINE ol8-19-rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNEWLSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 STABLE
2 ONLINE ONLINE ol8-19-rac2 STABLE
ora.CRS.dg(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 STABLE
2 ONLINE ONLINE ol8-19-rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 STABLE
2 ONLINE ONLINE ol8-19-rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ol8-19-rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 STABLE
2 ONLINE ONLINE ol8-19-rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 Started,STABLE
2 ONLINE ONLINE ol8-19-rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE ol8-19-rac1 STABLE
2 OFFLINE OFFLINE STABLE
ora.cdbrac.db
1 ONLINE ONLINE ol8-19-rac1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE ol8-19-rac2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.cvu
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.ol8-19-rac1.vip
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.ol8-19-rac2.vip
1 ONLINE ONLINE ol8-19-rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.scan1.vip
1 ONLINE ONLINE ol8-19-rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE ol8-19-rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE ol8-19-rac1 STABLE
--------------------------------------------------------------------------------
In my case, the ASM Network was not running on node 2, so I started it manually, and I will check in future reboots if I face the same issue again.
[[email protected] tmp]$ srvctl start asmnetwork
[[email protected] tmp]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1
Since everything is running smoothly, let’s remove the previous network from the Oracle CRS. As the grid user, proceed as below.
[[email protected] tmp]$ oifcfg delif -global eth2/10.1.2.0
[[email protected] tmp]$ oifcfg getif
eth1 192.168.56.0 global public
eth3 10.1.3.0 global cluster_interconnect,asm
In the next article, I will explain how to update the IP from the other layers of the Oracle CRS.
That’s it!