Database Access Problems through PIX

Hello Kevin,

the reason for this behaviour is indeed very likely to be related to the configuration of the PIX. The default TCP connection timeout is 60 minutes (command 'timeout conn'), and the maximum number of connections (which defaults to 0, which means unlimited), which specifies the maximum number of simultaneous TCP and UDP connections for the entire subnet may have been altered (command 'static (inside,outside) netmask 255.255.255.255 ).

Regards,

Yes, that is possible. It is also possible that the firewall has a configured "embryonic limit" or "connection limit" on the static.

The PIX must be at running either PIX 6.2 or PIX 6.3 software in order for that message to have been generated.

There would seem to be missing messages between the previous message and this one. Your snippet shows no examples of Built Translation, Teardown Translation, or Built Connection, and shows no IDS messages.

We do, though, see a Teardown Connection message, and we see that all of the listed messages are PIX-6- indicating that generally level 6 messages ("debugging") are being logged. The missing Built Connection message would be level 6 and so we would expect to see it here. There is the possibility of "log disable" attached to an access-list line that would have supressed the message, but if that had been done then we would not have seen the PIX-6-106100 message.

The Built Translation and Teardown Translation messages would not necessarily occur, if the destination IPs happened to be the targets of "nat (inside) 0 access-list", and there are some circumstances under which the translations would rarely be rebuilt (i.e, they sometimes activate and stay active for days, in some configurations). The lack of those two messages is thus not a reliable diagnostic of problems. If we look at the IP addresses involved, we see that both source and destination are in private IP ranges, suggesting that this traffic is within a VPN tunnel -- and for traffic within a VPN tunnel, a configuration that uses "nat (inside) 0 access-list" is much more common than not, so the lack of these two messages is probably not a concern.

The IDS messages are enabled by default, but only appear if the PIX thinks there has been a problem. A syn throttle would be sufficient reason. There are two different ways to disable the IDS messages, one by adjusting the "ip audit" parameters, and the other by turning off individual messages via "no logging message". Some of the signatures occur so often these days that individually disabling them is commonplace, but it would not be common to disable the ones dealing with syn floods.

If a syn flood was not detected by the connection limit had been reached, then IDS messages might not be generated, but other messages would have been generated.

The Translation messages and IDS messages and some of the other relevant messages are at different logging levels (mostly level 4), but only level 6 messages show up. With logging set at level 6, level 4 messages would be included (unless turned off). One possibility is that your syslog is configured to log the different levels to different files; if so then there would be additional messages available somewhere. But it still doesn't explain the lack of Built Connection message.

So.. the snippet you included is missing at least one important log message that should have been there unless it was explicitly disabled, and the log you are looking at might only be showing some of the available messages.

Perhaps you are not seeing it because it happend long before and the op missed it. Oracle clients are used to open a permanent control connection to the server, which lasts all the time the client is up and can be blocked by the pix in case of inactivity. If this is the case, the solution is however to increment the tcp inactivity timeout on the pix or, better, to implement some kind of keepalive on the database client. Bye, Tosh.

Many thanks to everyone who has responded! In regards to Tosh's comment,"If this is the case, the solution is however to increment the tcp inactivity timeout on the pix or, better, to implement some kind of keepalive on the database client. "

The customer said that indeed the firewall will timeout inactive connections after 30 minutes, but how does the firewall define inactivity? In this case we are actively executing queries against the DB and the majority of them succeed. It's just that often while we are actively executing these queries, the firewall suddenely starts denying the connection as you see in the log. In other words, the connection certainly wasn't "inactive" in the traditional sense as queries are being issued at least once every couple of seconds.

thx again so very much for your help here. I do appreciate it very much! Kevin

I just reported an issue I saw many times, it seems you are not facing the same, as far as you are sure to not reach the inactivity timeout. Keep in mind that if you have many sessions opened at a time each one has its own timeout....never mind if the others are very busy, if one of them is idle for 30 minutes the pix will shut it off. Again, if this is not the case you are facing a different kind of problem, maybe a resource exhaustion, a max limit of embrionic connections reached, a license issue, a pix bug, a client bug, a ..... Bye, Tosh.

Hi Walter

Thank you very much for your feedback on this issue. I very much appreciate it. I wanted to post an update of some new observations to see if maybe this will trigger some other ideas from anybody.

In the original PIX log I posted where as we are actively making database requests PIX suddenely starts denying our connections, we had database pooling disabled. That is for each and every database request we were opening a brand new connection to the database (or at least trying too).

After further testing we found that if pooling was enabled and we were maintaining 15 or more active connections in our pool, we would see similar behavior as above, going along fine executing successful DB queries one after another then suddenely have problems. However, instead of PIX suddenely starting to deny our (new) DB connections, the database request would go through on the already established connection, but the response from the DB would be held up for 10 - 15 minutes. We would never actually get the deny connection message, but the firewall did seem to be interferring with the return of results almost as if it was throttling traffic though nothing appears in the PIX logs according to the customer.

If we go back to not pooling DB connections but instead make a new connection for every single DB request, we see the original problem where we start getting DENY from PIX. We can now reproduce these two behaviors but have no solution and unfortunately the customer isn't helping much and says its up to us to figure out. :-(

Maybe this rings a bell with someone. If so, I would be very grateful to hear your ideas.

again, thx in advance!