SMTP and tcp ports

Matching SMTP exchanges with external SMTP servers (source port 25). This ACE would not match clients (source port >1023).

This ACE currently matches clients (destination port 25). Servers were matched on the previous ACE.

If you eliminate ACE # 50, clients and servers would match ACE # 60.

Best Regards, News Reader

You have static NAT setup for the SMTP server don't you?

e.g.: ip nat inside source static tcp 192.168.0.20 25 interface 25

That port is being committed for that purpose.

Any inbound connection setup would be directed at port 25.

Outbound SMTP connections from your server to an Internet-residing server would be from source port 25 with returning traffic coming to destination port 25.

With regard to random PAT ports, consider the following:

An internal client initiates a connection with source port 1200 to a server on the web. The packet is forwarded by the NAT router with source port 1200.

A second client initiates a connection with source port 1200 (coincidence) to any resource on the web. The NAT router will forward the packet with a "random unused source port" NOT involved in a pre-existing translation.

A random port is used when the desired source port is already being translated.

The random source ports (destination ports on the return path) aid PAT in associating returning packets with the correct internal host.

Best Regards, News Reader

Are you saying that the way servers assign the tcp ports is different from the way clients assign the tcp ports?

I thought that in either case, the host that initiates the session uses a random source port and a well known destination port with return packets coming back to the well known source port and a destination port matching the originating packet.

Yes, but NAT is handled by the edge router, which is managed by the ISP, so I can't see exactly what is going on in that box.

Thank you. This was helpful, but I still have a bit of confusion about source and destination well-known ports that I mentioned in the previous reply. Can you help me clear this up?

No, I don't think I have suggested that.

The original source port becomes the destination port on the return path, and visa versa.

You need to understand the operational behavior of the protocols you are using. Source ports are NOT always random.

e.g.: A Windows client initiates an NTP (or perhaps SNTP) connection to an NTP server on the router:

The host uses a "source port >1023", and a destination port of 123.

A Cisco switch initiates an NTP connection to an NTP server on the router:

The switch uses a "source port of 123", and a destination port of 123.

I was suggesting:

• That an external SMTP server initiating a connection to your server would likely be using TCP source port 25 which matches your ACE #50 (permit tcp any eq smtp host 192.168.0.20).

• That an external SMTP client initiating a connection to your server would be using TCP source port >1023, and would NOT match ACE #50 because you have specified the TCP source port 25 (smtp) requirement.

• That Both of the prior described connection attempts would match ACE #60 (permit tcp any host 192.168.0.20 eq smtp log) because "only" the TCP destination port 25 (smtp) has been specified in the ACE, and is common to both connection attempts. However, because inbound connections from SMTP servers matched ACE #50, the packets are forwarded without being compared to ACE #60.

Since ACE #60 would match both connection types, you could eliminate ACE #50.

Best Regards, News Reader

The information you provided lead me to believe that you were successfully connecting to your internal SMTP server from outside clients and servers. Correct me if I am wrong.

Has the ISP provided you with a pool of IP addresses, and are they performing static one-to-one translation with your internal hosts? If so, you would want to take steps to minimize the ports available on the SMTP server from outside hosts. A one-to-one mapping would expose all ports on the SMTP server.

With an edge router managed by the ISP, I would assume that Port Address Translation (PAT) is not being used. PAT (NAT overload) translates all (or a subset of) internal addresses to a single external IP address. In such a scenario, you would need to establish a port specific translation such as the one above, which would forward inbound connections to TCP port 25, to the internal SMTP server.

PAT can be used along side static and dynamic NAT pools as well, but it is doubtful that this scenario would exist without your knowledge/involvement).

Please see my reply to the other post.

Best Regards, News Reader

My "show run" from yesterday morning includes: access-list 102 permit tcp any host 192.168.0.20 eq smtp which is part of an ACL applied to the outside interface inbound.

Can you come up with an explanation for why this did not work?

The customers called me because they were not getting their mail so I added the other ACE with source port eq smtp. If I placed the new statement AFTER the old statement would it have received no hits?

We were trying to arrive at a plausible explanation for why you were seeing the hits on the two ACE's, without specific knowledge of your environment.

Note that I said - an external SMTP server initiating a connection to your server would "likely" be using TCP source port 25. It would be best to prove this belief (with a sniffer). I don't have an internal SMTP server here to verify it.

This is the first time you have mentioned customers, and you haven't described the scenario.

Are your customers on the inside or outside of the infrastructure you've described?

Is the internal SMTP server the one they connect to with their mail clients for sending mail, or is it just in the path?

You haven't conveyed whether the issue resides with connection initiation or the return path; whether we are talking about client/server, server/server, and initiated from where to where.

What SMTP functionality is working?

Statements like the following would help:

e.g.: An internal client sends mail via the internal SMTP server and that mail is successfully delivered to the destination SMTP server.

e.g.: An external SMTP mail server can not deliver mail to the internal SMTP server.

Best Regards, News Reader

Although I would still like and benefit from the clarification asked for in my other response, I'll offer the following possible theory:

Imagine that your internal SMTP server was making an outbound connection to an external SMTP server, AND that it actually uses a TCP source port >1023, AND that you did NOT have "inspection" of this traffic configured on the router (to facilitate the return path).

The returning traffic would be TCP source port 25, and TCP destination port >1023.

ACE#50 (permit tcp any eq smtp host 192.168.0.20) would then be required in the absence of correct "inspection".

Best Regards, News Reader

I'm not sure which you are referring to as old vs. new.

Since they are both permit ACEs, with no deny ACE between them, changing the order of the ACEs would only change the order in which they were evaluated against inbound packets. The functional results would be the same.

However, it would have been beneficial for you to have changed the order of the ACEs first, and monitored the resulting matches, rather than deleting ACE #50.

Had you done so, matches on the resequenced ACE would have demonstrated that both ACEs were required to retain functionality, and your customer would not have experienced a service interruption.

Best Regards, News Reader

We've got two sub-threads going here which makes this a bit ackward. I will avoid this in the future. In addition, I believe I reported incorrectly earlier. Please let me try again.

Two days ago, the only smtp ACE was "access-list 102 permit tcp any host 192.168.0.20 eq smtp". With this, we were able to receive mail but could not send mail out.

Yesterday, I removed that statement and replaced it with "access-list

102 permit tcp any eq smtp host 192.168.0.20". I can't recall whether this resulted in not being able to send mail or not being able to receive mail but it was one of these. When I added "access-list 102 permit tcp any host 192.168.0.20 eq smtp log" everything worked.

If you don't mind, please explain why the various ACEs blocked either inbound or outbound mail. What is the simplest safe way to set up the ACL properly?

There are two MX records. Does that mean that *all* incoming mail will be sourced from one of these hosts?

Will outbound mail exchange sessions always be with *only* the ISP's smtp server?

We have a block of 16 public IP addresses. About half of these are mapped to inside servers, including the Exchange server. I understand the issue you raised about all the ports being open. This is why I want to implement a better ACL.

In addition to the static translations, the ISP configured "ip nat inside source list 1 NAME overload" for all the PCs on the inside. We are doing no translation on our routers.

In spite of saying that I would try to avoid two sub-threads, you've raised a number of important questions that I will attempt to answer here and in your next two replies in this sub-thread.

In order to simplify this discussion I will refer to all parties desiring to send and receive email as "us".

The Outlook client is configured to use the local Exchange server for mail.

Yes. I'm not as knowledgeable about smtp as I need to be to correctly analyse or even to report all the relevant issues. If I were, I would have created a working ACL without trial and error. Please refer to info I left in the other sub-thread.

ditto

I see that additional analysis is needed. Thank you for pointing the way.

Inspection is another unresolved issue. Inspection was enabled but we were having browsing problems that two Cisco TAC engineers were unable to resolve. As an experiment, I turned off inspection and we were able to browse again. That is when I started developing the ACL.

This makes perfect sense. I knew there had to be a logical explanation for what I observed.

In this context, I meant the ACE with seq # 50.

What I meant was, since you said that I only needed seq # 60, if it were first then 50 would get no hits. Isn't that correct?

I understand. Thanks.

One of your paragraphs below requests interpretation of the ACLs so I'll do so. As you stated earlier, ACL 102 is applied inbound on the external interface (correct me if I am wrong).

A modest (no embellishments) extended ACL takes the form:

action, protocol, source, qualifier source-port, destination, qualifier destination-port

If you leave out the source or destination ports and their qualifiers, those criteria are not used in evaluating a match.

permit tcp any host 192.168.0.20 eq smtp

The ACE above (lets call it ACE #1) "permits" "any external host" (due to where you applied the ACL), with any source port (because you haven't specified one), to connect to "host 192.168.0.20" on "TCP destination port 25" (i.e.: specified with smtp keyword).

This ACL would permit access to the internal SMTP server (listening on TCP port 25) from external clients and servers. The mail clients would be using a TCP source port >1023, and external mail servers would be using TCP source port 25, or TCP source port >1023 (I'm not sure which, but would have expected TCP source port 25). Both would get through because your ACE is not evaluating the source port.

You were successful in receiving mail from outside sources because ACL

102 permitted the inbound connection, and you addressed return path requirements (perhaps with ACLs, or with Inspection).

You removed the statement that facilitated inbound connections to the SMTP server from "any source port".

... and replaced it with an ACE (lets call it ACE #2) that required a "source port of 25".

This ACE prevents e-mail clients from accessing the server (they use source port >1023). This ACE may or may not prevent external SMTP servers from accessing the server depending on the answer to our earlier question (i.e.: does the external server use source port 25, or a port >1023). See below for a determination.

More on ACE #2 to follow below.

ACE #1 restored your ability to receive inbound connections "initiated" from both external clients and servers (no source port requirement defined).

If both ACLs were required for full functionality, that suggests that ACE #2 is impacting outbound mail. Since this ACE is applied inbound on the outside interface, this suggests that the ACE is addressing the return path for outbound connections. This suggest that you are not utilizing "inspection" to your advantage.

permit tcp any eq smtp host 192.168.0.20

ACE #2 is matching connections with a "TCP source port 25" (i.e.: an external SMTP server). This affirms my belief that server-to-server exchanges would occur port 25, to port 25.

The absence of ACE #2 would prevent your internal server from successfully establishing a TCP connection with an external SMTP server.

I would create an ACL on the inside interface to facilitate outbound connections, and use inspection to facilitate the return path. Similarly, I would create an ACL on the outside interface for inbound connections, and use inspection to facilitate the return path.

1. Get to know your applications (i.e. the source and destination ports used).
2. Start with connection initiation from a client with a specific protocol/application, and provision for the outbound connection with ACLs.
3. Address the return path (typically provisioning with inspection).
4. Are clients on both the internal and external interfaces? If so provision for each. You need to examine all scenarios to get it under control.
5. Move on to the next protocol.
6. etc.

You should document what you know about each protocol and its requirements, and how each protocol is provisioned.

Sourced?

If the MX records are under your domain, I believe they would convey the IP addresses of your "receiving" SMTP servers to those who query for this info.

Depends on the configuration. It is conceivable that it could be configured to forward mail through an ISP's SMTP server on its way to final destination. It is also conceivable that your server would forward to the recipients SMTP server. You'd have to examine the configuration of the server.

Best Regards, News Reader

What we have done, and what I would recommend is the following.

Setup a sniffer, and examine traffic for each of the protocols that your organization relies on. This will educate you about what source and destinations ports are used by the clients and servers. Record what you learn, it is the basis for your implementation strategy.

Document your requirements for a given protocol, address those requirements, then move on to the next protocol.

e.g.: internal clients using protocol "x" will initiate connects to external server.

Create the ACE(s) required to support the outbound connection. My preference is to apply the ACL that will contain them, inbound, on the internal interface. Then consider the return path. It is preferable to use "inspection" rather than ACLs. Again, my preference is to apply inspection inbound on the internal interface. This will provision the return path (i.e.: temporary dynamic holes inbound on the external interface).

For connections coming from the Internet, I would apply an ACL inbound on the external interface, along with inspection applied inbound.

Every protocol that is not natively supported by IOS inspection can still be inspected with Granular Protocol Inspection. You would use Port to Application Mapping to associate a user-defined name with a port or group of ports, then create an inspection rule specifying the user-defined name.

With this approach you can eliminate the use of generic TCP and UDP inspections, providing a more secure implementation.

Be cautious with the HTTP inspection, it can result in heavy CPU utilization if not configured properly. Perhaps this relates to your "browsing problems". We avoid this method of inspecting HTTP. You would be better off using the APPFW (Application Firewall) for HTTP inspection if you are not needing Java Blocking and URL filtering. APPFW configures policy for HTTP inspection (e.g.: compliance with RFCs, protection from IM, P2P, and tunneling), and is then included in your inspection rules.

With our IPSec + GRE tunnels we also use inspection on tunnel interfaces.

You should be using auditing with syslog to gain visibility on what is happening on your interfaces.

ip inspect log drop-pkt ip inspect audit-trail

Likewise, competency with a sniffer will reduce time-to-resolution for many issues.

Using a uniform approach to provisioning protocols, and documenting each is important. A document that says this, this and this are required to support application X, is valuable when trying to diagnose an issue.

Best Regards, News Reader

Bob:

Perhaps you could place the following ACE "above" the other two temporarily, to confirm the presence or absence of port 25-to-port 25 traffic. This would help us end the guessing.

102 permit tcp any eq smtp host 192.168.0.20 eq smtp log

If you were using inspection with auditing, we'd see exactly what we needed to know in your logs (source and destination IPs, protocol, and ports). But, we'll settle for matching the ACE at the moment.

Do you have users accessing the server from the outside?

Best Regards, News Reader

Thanks. I added the statement and there are no hits after 20 minutes. I will look later today or tomorrow morning. Do either mail clients or servers normally set both source port and dest port to 25? If not and if we see this, what is the significance?

I have used debug ip packet acl detail to see this but it doesn't always capture the packets even when a matching ACE increments. Is this because the packet was CEF switched rather than process switched?

Yes, they can get their mail remotely via their web server. I am not involved in this and can't tell you how it works other than to say that I had to add an ACE for both 80 and 443 inbound to the server to enable it.

The direction of communication provides context. The SMTP client opens a random TCP port >1023 prior to initiating a connection to the server. The SMTP server is actively listening on "well-known" TCP port 25.

A "packet" sent from the client to the server has a TCP source port >1023, and a TCP destination port of 25 (server listening on port 25).

The returning "packet" has a TCP source port of 25, and a TCP destination port >1023 (client listening on specific port >1023).

For a server-to-server connection, we know that the server being connected too, will be listening on TCP port 25 (destination port in this context). We are using the newly introduced ACE to determine whether the server "initiating" the connection is using a source port >1023, or a source port of 25.

If you know server-to-server connections are occurring during the monitoring period, and we see no matches (102 permit tcp any eq smtp host 192.168.0.20 eq smtp log) we will conclude that the server initiates connections with a source port >1023.

We'll at least learn something, and although not necessary, we could refine the ACEs further.

For future reference:

You were using the following ACE to facilitate inbound connections to the SMTP server from "any source port":

102 permit tcp any host 192.168.0.20 eq smtp

You were using the following ACE to facilitate traffic returning from external servers (due to the absence of inspection):

102 permit tcp any eq smtp host 192.168.0.20

I don't know.

As an aside, investigate Fast Switching, if you have not already done so.

interface FastEthernet0 ip route-cache flow

There are other commands like it worth investigating as well.

Best Regards, News Reader