Troubleshooting gc block lost and Poor Network Performance in a RAC Environment

Environment:

• Oracle Linux
• Red Hat Linux
• Oracle 11G
• Oracle 12C

Symptoms:

High cluster wait and performance loss due to cluster wait.

Diagnose:

[root@node2 ~]# netstat -s |grep assem
    1169750855 reassemblies required
    203909271 packets reassembled ok
    3180797 packet reassembles failed

 

and countinuesly increases.

 

Monitor reassembly failure with script:

 

 

[root@node2 ~]# cat packages.sh 

while true;

do

  x=`netstat -s | grep "packet reassembles failed"|awk '{print $1}'`

  echo `date` ": " `expr $x - $t`;

  t=$x

  sleep 5;

done;

 

[root@node2 ~]# ./packages.sh 

Pzt May 6 14:57:08 +03 2019 :  126

Pzt May 6 14:57:13 +03 2019 :  132

Pzt May 6 14:57:18 +03 2019 :  186

Pzt May 6 14:57:23 +03 2019 :  152

^C[root@node2 ~]# 

 

[root@trakyatipdbnode2 ~]# sysctl -a|grep ipfrag

net.ipv4.ipfrag_high_thresh = 4194304

net.ipv4.ipfrag_low_thresh = 3145728

net.ipv4.ipfrag_max_dist = 64

net.ipv4.ipfrag_secret_interval = 0

net.ipv4.ipfrag_time = 30

 

Solution:

Default value for buffer usage during IP packet reassembly 4M for high 3M for low threashold. Problem is solved after incrementing to 15-16M values.

[root@node2 ~]# echo "16777216" > /proc/sys/net/ipv4/ipfrag_high_thresh
[root@node2 ~]# echo "15728640" > /proc/sys/net/ipv4/ipfrag_low_thresh

To make this change permanent across reboots, edit the /etc/sysctl.conf file and add the line:

net.ipv4.ipfrag_high_threshold = 16777216
net.ipv4.ipfrag_low_thresh = 15728640

Immediately after setting new kernel values, cluster wait value drops to 0 and "netstat -s |grep assem" shows there is no change in packet reassembles failed.