Firewall timeout and TCP RST

- to kick someones ass for being so idiotic

Yours, VB.

Does it do it through a VPN connection?

If not, then it's a risk and should be rethought.

Remote databases should be access through some form of tunnel and not directly exposed to the world. That would solve your problem.

Thank you for your interest in the security aspect of the question, but this was not the topic of my post (I'm not the security designer!)

I'm looking for the best method to work-around the problem from the source-code point of view. I cannot nor want change anything in the topology of the network. However: clients and DBs are inside a 10.x.x.x network that is unaccessible (!) from the outside world. The FW is intended as part of a 'segmentation-policy' of the enterprise network.

I add that the FW doesnt require authentication.

bye n.dzl

Then you're looking at it from the wrong side.

Once you've been given access through a tunnel then you don't have to worry about time-outs.

You clearly said that the DB's were protected by a firewall and that you access them remotely.

If the firewall is setup properly and you have a secure tunnel, there is no timeout and you don't need to change your solution.

So, either you didn't write your program to close and open the connection as needed, which is a bad move, or you don't have a secure tunnel and proper firewall setup.

So, with that being the case, have you asked the firewall controller about the timeout?

Have you considered properly coding your app to open/close connections as needed instead of leaving the connection open all the time?

...

ok, I'll try to explain myself better.

TODAY (03/oct/2007) the status is:

- a 10. network

- many clients

- many DBs (in a wide-sense)

WITHIN FEW WEEKS (>01/nov/2007) situation will be:

- a 10. network

- many clients

- a FW between clients and DBs, with Timeout on low traffic connections and no authentication

- many DBs (in a wide-sense)

Today everything works fine. I'm evaluating which is the impact for those clients which firstly open a connection and use it until it is available (and generally try to reconnect). The case may be that of CORBA clients or ORACLE clients. Here the connection cannot be done every time I query the DB: it would be extremly slow, while the system must answer 'in real time'.

Please don't say me "CORBA is not suitable for ral time system" or sth like this for ORACLE...! I'm a programmer from the client side. Server side is not under my control! The same for the FW: I must work with it from november; Nothing if its configuration etc.. is under my control. I simply must comply with the enterprise rules!

bye n.dzl

Again, this would seem to be simple, you query for the connection and if it fails you reconnect.

While I can't understand why the problem, you need to be able to talk/address the issue and you need to understand that maintaining a remote connection is problematic and you should have accounted for it in your application.

The far easiest solution, is to get the firewall administrator to adjust the timeout on the database port. This is trivial with just about every firewall. The cleanest solution imo, would be that your application are able to reconnect when needed, if the connection has been reset. What happens with all your clients now, if the db for some reason suddenly restart?

I have seen several applications doing a query like "select * from dual" (on oracle) every x minute, to keep a connection open. Wouldn't say it's elegant, but it works.

Erik

Hi!

I'm working for a huge company. When 'they' decide that FW has a timeout of N minutes, they carve it on the rocks!

Right. And obviuosly I was thinking about it. But I wonder if a drop-down from a FW is seen from the client in the same way as it were a server not responding. Just a detail: if server is down it is meaningful to try connection after some seconds, while if the lock comes from the FW the client should retry faster. In fact I need to understand who is locking the connection...

That was what I wanted to read! I think I'll do the same!

Thank you Erik. Happy coding. Nik.

Only noobs code that way, it's always a bad move. If you are not able to detect when you've lost your connection then you're not coding properly. Don't write crap code to just make your life easier.

Well, any number of things, in no particular order, some perhaps better than others.

*) emit a message to the user upon reciept of the RST and require them to interact with the program to establish a fresh connection to the DB. Include in the message you present a suggestion that this may be the result of firewall settings. *) simply re-establish the connection when it drops with the RST. you probably already need to be doing something like that to handle "normal" drop cases anyway. *) implement an application-level keepalive mechanism with an initial setting of say 1/10th the firewall timeout to make things robust as the firwall starts to get overloaded and the administrator of the firewall makes the local-to-the-firewall optimization of lowering the timeout. I think someone else has suggested some sort of "noop" DB query. Perhaps there is something else. Be prepared to direct the admin(s) of the DB system to the firewall folks when the load on the DB system increases. *) start to depend on the clients systems' setting for TCP Keepalive intervals being sufficiently less than the firewall's timeout and set SO_KEEPALIVE on your sockets connected to the DB. Frankly, while this is the most facile thing to do, it is IMO the biggest kludge because it is depending on the TCP stack being configured with a keepalive interval that works with the firewall's timeout. it is also possible for the firewall to see that this is a TCP keepalive and perhaps still decide that the connection is idle.

rick jones

You'll find that this is a very common problem, and pretty much the only two reasonable ways around it is to either do an application level keep-alive, or stop having long idle sessions (IOW connect- process-disconnect instead of connect-process-wait-process-wait...). Other methods can work, but tend to be much more painful in the long run, both in terms of reliability and management, although if this is a single installation, they may be worth considering.

Aren't they answerable to management, who has an overriding goal to allow work to get done? Firewall administrators can't bring the company's business to a halt.

If the firewall sends a RST, it looks similar to the server crashing and rebooting. The next time the client sends a packet, the server will respond with RST because the connection no longer exists.

Why do you need to retry faster? If the client is idle, what difference does it make whether the connection is up or down? Re-establish the connection the next time you need to access the DB.

One of the reasons firewalls have timeouts like this is because connections use up resources in the firewall's state table. Without timeouts, all the idle connections fill up the table. So you shouldn't create connections that aren't actually needed.

[...]

I'm not sure exactly what you mean by that, but several database libraries I use are able to automatically reconnect if the connection is lost for any reason. If it's possible, that's a nice solution that fixes the problem at hand and also makes your application more robust in the face of other failures.

Also, for a quick fix, sometimes setting TCP keepalives with a fairly low timeout will keep these firewalls from timing out connections.

Good luck!

----Scott.