Hello,
I am currently working to diagnose an issue with a Nokia IP330 running Checkpoint NG. The issue is particularly odd, I have never seen issues like that and I am stumped as to what the root cause of the problem is so that I can work to resolve it.
Currently the network monitoring server is outside our management firewall. When the system is located outside the firewall we do not have any network related performance problems to the monitoring server. However when we move the network monitoring server to behind the Nokia IP330 server we have terrible network performance to the monitoring server.
When I say terrible network performance I am seeing the following symptoms.
- packet loss, ICMP loss of around 85% or higher.
- very jumpy connection, and very very lagged response times.
- unable to establish a new ssh connection to the server for some time, then it will finally connect after a several minute delay.
Normally I would see those types of symptoms if the network connectivity and/or system load was very very high. However that is not the case. I can go and look at the system load on the monitoring server and I see that it is only around 5-15% CPU, 80% ram, iostat reports relatively low disk i/o and there is no I/O wait in top. If I check the network utilisation on the Cisco Catalyst 6509 (Running CatOS) interface its only around 1% (1Mbps). In all cases the interfaces are running at 100Mbps Full Duplex.
I am getting no errors on the network interfaces on either the Server, Switch, Switch to Checkpoint, Checkpoint to router or anywhere else on the network. So i do not think that its a network related issue as far as the switching and routing infrastructure is concerned.
The network performance only shows up on the monitoring server other systems on the same switch, subnet, vlan and checkpoint firewall interface see no performance issues. The interface on the checkpoint is on an expansion board and the checkpoint has a rule to pass all traffic from the monitoring server to the network subnets for monitoring. The problem only shows up when the ICMP monitoring is enabled, the service monitoring works fine without causing any performance problems. We have no icmp rate limits set on any of the switches either.
If I check the checkpoint system the utilisation is very low, the system does not go above 50% utilisation across the board during the icmp monitoring poller. if we move the server so that it is stand-alone behind a stand-alone checkpoint / Nokia IP330 the system does not show the same symptoms.
The NIC cards on the Server are Broadcom 10/100/1000 copper cards running with the boardcom driver on Redhat Enterprise Linux v3 update
Has anyone seen any sort of similar issues, or does anyone have any advice as to what I should be looking at. I am not 100% on the Checkpoints so any advice would be great as I am sure that is the root cause of the problem. I am just unable to conclusively prove it either way.
Regards,
Johhny