Nokia IP330 / Checkpoing NG

Hello,

I am currently working to diagnose an issue with a Nokia IP330 running Checkpoint NG. The issue is particularly odd, I have never seen issues like that and I am stumped as to what the root cause of the problem is so that I can work to resolve it.

Currently the network monitoring server is outside our management firewall. When the system is located outside the firewall we do not have any network related performance problems to the monitoring server. However when we move the network monitoring server to behind the Nokia IP330 server we have terrible network performance to the monitoring server.

When I say terrible network performance I am seeing the following symptoms.

- packet loss, ICMP loss of around 85% or higher.

- very jumpy connection, and very very lagged response times.

- unable to establish a new ssh connection to the server for some time, then it will finally connect after a several minute delay.

Normally I would see those types of symptoms if the network connectivity and/or system load was very very high. However that is not the case. I can go and look at the system load on the monitoring server and I see that it is only around 5-15% CPU, 80% ram, iostat reports relatively low disk i/o and there is no I/O wait in top. If I check the network utilisation on the Cisco Catalyst 6509 (Running CatOS) interface its only around 1% (1Mbps). In all cases the interfaces are running at 100Mbps Full Duplex.

I am getting no errors on the network interfaces on either the Server, Switch, Switch to Checkpoint, Checkpoint to router or anywhere else on the network. So i do not think that its a network related issue as far as the switching and routing infrastructure is concerned.

The network performance only shows up on the monitoring server other systems on the same switch, subnet, vlan and checkpoint firewall interface see no performance issues. The interface on the checkpoint is on an expansion board and the checkpoint has a rule to pass all traffic from the monitoring server to the network subnets for monitoring. The problem only shows up when the ICMP monitoring is enabled, the service monitoring works fine without causing any performance problems. We have no icmp rate limits set on any of the switches either.

If I check the checkpoint system the utilisation is very low, the system does not go above 50% utilisation across the board during the icmp monitoring poller. if we move the server so that it is stand-alone behind a stand-alone checkpoint / Nokia IP330 the system does not show the same symptoms.

The NIC cards on the Server are Broadcom 10/100/1000 copper cards running with the boardcom driver on Redhat Enterprise Linux v3 update

  1. Has anyone seen any sort of similar issues, or does anyone have any advice as to what I should be looking at. I am not 100% on the Checkpoints so any advice would be great as I am sure that is the root cause of the problem. I am just unable to conclusively prove it either way.

Regards,

Johhny

Reply to
exter_c
Loading thread data ...

How much memory on the 330 ?

greg

Reply to
Greg Hennessy

vmstat 1 procs memory page faults cpu r b w avm fre flt re pi po fr sr w0 in sy cs us sy id 0 0 0 566440 54092 3 0 0 0 5 2 3 67 133 22 7 45

48 0 0 0 566440 54076 1 0 0 0 0 0 0 2013 685 23 1 48 52 0 0 0 566440 54056 1 0 0 0 0 0 0 2084 563 19 0 49 51 0 0 0 566440 54020 1 0 0 0 0 0 0 1866 632 22 1 34 65 0 0 0 566440 53932 1 0 0 0 0 0 0 1944 536 20 0 44 56
Reply to
Johhny

I meant physical memory.

greg

Reply to
Greg Hennessy

Hello,

sorry use to another version of the vmstat utility that reports both ram and vm. We have 256meg of physical ram.

Regards,

Johhny

Reply to
Johhny

Re-create the problem and run "fw monitor -o test.cap" on the Nokia. Copy the file "test.cap" that is generated and examine it in detail using ethereal (or, from Checkpoint, cpethereal). The fw monitor allows you to see what is happening in the Checkpoint virtual machine, you can see packets being examined as they enter the kernel - pre-inbound (i), post-inbound (I), pre-outbound (o) and post-outbound (O). This at least will show where and if the packets are being inspected and where they may be being dropped or delayed or whatever.

Wayne McGlinn Brisbane, Oz

Reply to
Wayne McGlinn

I'd consider 256meg to run NG in as being 'tight', especially on a 330 with a slow memory/disk subsystem.

greg

Reply to
Greg Hennessy

I understand from your description that you already verified the collisions rates on the Cisco Switch and that the problem occurs only with the monitoring server. If not, you can have a look at these statistics on the switch.

Have you analyzed the traffic with SmartTracker? Is the traffic always green (accepted)? Also check the SmartDefense configuration on the SmartCenter.

A.

Reply to
Adrian

Why? You dont know how many connections he has. Checkpoint doesn't need much memory.

Wayne McGl>

Reply to
Wayne McGlinn

NG is a hog when compared to the 4.0/4.1 the 330 was designed for.

Their suitability for running NG has been done to death on the wizards mailing list.

greg

Reply to
Greg Hennessy

I don't know about IP330, but if you try to run SPLAT NG in 256MB, it goes into swap thrash before it comes up far enough to accept a connection. Same box runs nicely with 512MB.

Triffid

Reply to
Triffid

Good points, both of you :) I'd forgotten how much more intensive NG and NGX is on those older Nokia's.

Wayne

Reply to
Wayne McGlinn

Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.