So I'm tossing this out there to see if anyone might have some thoughts on where to look because I just find this baffling.
We had a Cisco 3750 Catalyst switch set up with a number of hosts connected to it. All of these hosts were configured to do NIC teaming with etherchannels set up on the switch. We then added a second 3750 to create a stack. We re-configured the etherchannels on the existing servers so that each one had one connection to each switch. We then added a number of new servers with each one using etherchannels (one connection to each switch). Everything seemed fine. All of the servers were available on the network, etc. But, we noticed that of the 12 new servers, two of them had this odd issue. All of the servers are in the same VLAN and IP's in the same subnet. All of them can get out to the internet and can be reached by *most* computers in the same VLAN and in other VLANS. BUT...the two oddballs can't talk to the other servers connected to the same 3750.
I turned off the NIC teaming and disabled the secondary NIC port. We removed the switch ports from the etherchannel so it was just a normal, plain jane port. Still had the problem. Moving the server IP information from the primary to the secondary NIC port cleared up the problem...at first. While trying various permutations of the NIC teaming (trying to track down the problem) things got the point where if the server was configured to use NIC1, it could get out to the internet and other various servers, but couldn't talk (no ping, no thing..they don't even see ARP requests) to the other servers on the switch (EXCEPT those in a different VLAN) or some other random machines on that same VLAN.). Setting the server to use NIC2 resulted in it being able to talk to servers it previously couldn't talk to, but it suddenly couldn't talk to the gateway and so couldn't reach anything else. It was previously able to do so.
What this FEELS like is that the switch has somehow declared these two server's MAC addresses as persona non-grata and won't route traffic from or to it except to seemingly random exceptions.
Anyone have any thoughts on where to look as I'm pretty much stumped at this point.