FW1 NGX R62 on IP530 issue

Jan 08, 2007 5 Replies
G
FW1 NGX R62 on IP530 issue open original image

Hello,



We have a cluster of firewall checkpoint FW1 NGX R62 running on Nokia IP530 platforms. Since, I upgraded to R62 (from R55), every 3/4 days, the master firewall stop running correctly, it doesn't forward the traffic anymore but the hardware is still up. So the failover is not working. I have to switch to the backup one manually. Here, what can I see on the logs of the firewall when it fail:



Jan 7 01:42:51 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:42:55 egb [LOG_CRIT] kernel: cpas_pcb_connect: received NULL pcb as argument, failing Jan 7 01:42:55 egb [LOG_CRIT] kernel: cpas_pcb_connect: received NULL pcb as argument, failing Jan 7 01:42:58 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:42:58 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:42:58 egb [LOG_CRIT] kernel: cpas_pcb_connect: received NULL pcb as argument, failing Jan 7 01:44:01 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:44:06 egb last message repeated 3 times Jan 7 01:44:10 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (90, 0, 0)! Jan 7 01:44:10 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:44:13 egb [LOG_CRIT] kernel: cpas_tcp_input: failed to allocate cpas_pkt_buf for in-sequence data Jan 7 01:44:14 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:44:18 egb [LOG_CRIT] kernel: FW-1: fwtcpstr_init_stream: error allocating kbuf Jan 7 01:44:19 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:44:21 egb [LOG_CRIT] kernel: cpas_tcp_input: failed to allocate cpas_pkt_buf for in-sequence data Jan 7 01:44:26 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:46:25 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:46:25 egb last message repeated 5 times Jan 7 01:46:28 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (91, 0, 0)! Jan 7 01:46:28 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:46:28 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:46:31 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:46:31 egb [LOG_CRIT] kernel: FW-1: loghandle_create: failed to allocate memory for the log handle Jan 7 01:46:31 egb last message repeated 2 times Jan 7 01:46:31 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (91, 0, 0)! Jan 7 01:46:34 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:46:34 egb [LOG_CRIT] kernel: cpas_tcp_input: failed to allocate cpas_pkt_buf for in-sequence data Jan 7 01:46:35 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:46:37 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (91, 0, 0)! Jan 7 01:47:23 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:47:26 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:47:32 egb [LOG_CRIT] kernel: FW-1: loghandle_create: failed to allocate memory for the log handle Jan 7 01:47:32 egb [LOG_CRIT] kernel: FW-1: fw_xlate_packet: fw_xlate_log failed Jan 7 01:47:35 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (90, 0, 0)! Jan 7 01:47:35 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:47:38 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (90, 0, 0)! Jan 7 01:47:39 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:47:39 egb last message repeated 4 times Jan 7 01:47:45 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:47:46 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:47:58 egb last message repeated 4 times Jan 7 01:48:05 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (29, 0, 0)! Jan 7 01:48:05 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:48:43 egb last message repeated 2 times Jan 7 01:48:48 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (32, 11, 0)! Jan 7 01:48:48 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:48:48 egb last message repeated 2 times Jan 7 01:48:58 egb [LOG_CRIT] kernel: FW-1: loghandle_create: failed to allocate memory for the log handle Jan 7 01:48:58 egb [LOG_CRIT] kernel: Error: FW-1 failed to generate the log record. Jan 7 01:48:58 egb [LOG_CRIT] kernel: FW-1: fwx_create_xlbuf: error allocating kbuf Jan 7 01:48:58 egb [LOG_CRIT] kernel: FW-1: fwx_get_xlation: fwxl is NULL Jan 7 01:48:58 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:48:58 egb [LOG_CRIT] kernel: FW-1: fw_spii_new_h: failed to put data in kbuf. Jan 7 01:48:58 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:17 egb last message repeated 5 times Jan 7 01:49:20 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (29, 0, 0)! Jan 7 01:49:22 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:22 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:35 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:49:35 egb [LOG_CRIT] kernel: cpas_tcp_input: failed to allocate cpas_pkt_buf for in-sequence data Jan 7 01:49:36 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:38 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:49:38 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:38 egb [LOG_CRIT] kernel: FW-1: loghandle_create: failed to allocate memory for the log handle Jan 7 01:49:38 egb [LOG_CRIT] kernel: FW-1: fw_xlate_packet: fw_xlate_log failed Jan 7 01:49:38 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:49:38 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:49:38 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:39 egb last message repeated 2 times Jan 7 01:49:41 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (79, 0, 0)! Jan 7 01:49:41 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:41 egb [LOG_CRIT] kernel: FW-1: loghandle_create: failed to allocate memory for the log handle Jan 7 01:49:41 egb [LOG_CRIT] kernel: Error: FW-1 failed to generate the log record. Jan 7 01:49:41 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation Jan 7 01:49:44 egb [LOG_CRIT] kernel: fwhandle_get(fwseqvalid.c:914): Table kbufs - Null handle requested Jan 7 01:49:44 egb [LOG_CRIT] kernel: FW-1: fw_spii_add_inspection: error - failed to write message (4, 0, 0)! Jan 7 01:49:45 egb [LOG_CRIT] kernel: fwlock_call_at_release: no memory available for operation



If anyone have already see those error messages? I found nothing on internet and checkpoint KB. Thanks for your help...



Vote
Twitter Facebook LinkedIn
M

hi, have you checked this:

formatting link
do you have enough memory? M

Vote
Twitter Facebook LinkedIn
G

Hi,

Yes I'm on Nokia IP530 with 512MB of memory

Thanks

Vote
Twitter Facebook LinkedIn
R
1) Monitor your firewall (memory) usage during the 3/4 days while it's working fine using: #fw ctl pstat

You can download a script from the Nokia support site that does this automatically:

formatting link
(Nokia's solution id 11093)

2) When you switch to the standy firewall does this one also has the same problem after 3/4 days?

Br. Robby

Vote
Twitter Facebook LinkedIn
G

Thanks for your response

1) We have already a monitoring tool which monitor the firewalls. And I didn't see any peak or saturation of memory before the problem occurs.

2) Yes, the same problem occurs independently that it's the first or second firewall of the cluster which is master.

Regards David

Vote
Twitter Facebook LinkedIn
R

"fw ctl pstat" will show you more information then just general system memory usage. The Check Point kernel only uses a part of the available system memory. This part can be variable or a hard limit (in which case you might need to increase it.)

Do some research on the output of "fw ctl pstat" (hmem, kmem, failed alloc,...) . There's a lot of information in it. Looking at the errors you posted you'll probably find some answers in the output of it.

Br. Robby

Vote
Twitter Facebook LinkedIn

Join the Discussion

Have something to add? Share your thoughts — no account required.

Didn't find your answer?

Ask the community — no account required