Mystery "ghost" MAC addresses?

First thoughts....

The fact that the source MAC is "weird", the traffic is a broadcast, and is a full length packet (1510 bytes) leads me to believe that you have a device on your network that has a virus.

I did work on a problem more than 12 years ago where we had bridges (CrossComm ILAN's) that replicated a randomly generated packet, but this was due to a bug in the bridge software. I have been working with Cisco switches and routers for more than 10 years and I have never seen or heard of a problem like this with Cisco equipment. I would suspect the "other" vendors equipment before Cisco.

=A0

* One time I saw some similar traffic it turned out to be to do with network load balancing enabled on two NICs on two different servers. The core switch started reporting a flapping mac-address & sure enough, 1 second it would be on one uplink - to server A - and the next it would be on another - to server B. As neither server had dual NICs we just disabled the feature as it was unnecessary. Unfortunately, I do not remember the MAC address & whether it was b/ cast or other. All I remember is that it wasn't a registered OUI & I couldn't find anything about it online...

Well, let try this as a wild guess: if the switch itself emitted a malformed packet due to a bug, this could be the problem. The packet is pretty clearly malformed, looking only at the ethertype. And it is a maximum sized packet, which is suspicious for a broadcast that is not part of a stream. Some switch seems to have a pointer to memory which is hoarked.

Let's presume it came from the Dell for the moment. The Cisco switch will flood it (since it is a broadcast) and mark the interface where it heard the broadcast (Gi0/6) as the source. The packet will flood and the Dell will ultimately hear the packet back on that same port (depending on the state of the spanning tree or nature of the bug that caused this packet), and it will mark the interface where it heard it from. Since it has no way to know that it generated this malformed packet itself, it adds it to the MAC address forwaring table, presuming it must be a distant host.

This is how you get a "synthesized" packet in the middle of the wire - One of the two switches emitted a malformed broadcast packet and did not recognize the packet as its own when it came back.

The partial hex dump of the packet has no obvious clues (to me) about what it is. Sometimes, these are clearly legitimate packets shifted by a number of bytes. A more motivated person might play around with bitshifting the hex dump or looking at a more complete hex dump to see if any recognizable patterns emerge. But as near as I can tell, this is probably some kind of bug in one of the switches, and it seems harmless as long as these emissions are rare.

Are the MAC address entries timestamped? If they are timestamped with high precision and the switches are synchronized, the one with the oldest MAC address entry is probably the innocent one (since that switch heard the impossible packet first).

Does the packet appear regularly? If you can somehow diagnose which side "shoots first" you will be closer to understanding where it comes from.

that Ethertype 886f "belongs" to Microsoft. AFAIR it is something to do with server load balancing.

i think when this is used the server uses a different source address - but it makes it up from the NIC address.

Look to see if 1 of the source adr MACs on the server is the same apart from a couple of the high order bits.

Well, I guess you can disregard my last post, then! :-)

With your clue, I Googled some more: Google "0x886f load balanc Microsoft's Windows NT Load Balancing Service (WLBS), a clustering service. Packets sent using this Ethertype have the following properties:

a.. The destination MAC address is either the broadcast (FFFFFFFFFFFF) or a multicast address that is set in the registry. b.. The source MAC address is 00 BF followed by the IP address of the machine sending the packet. That machine also has a corresponding MAC address of 02 BF ... that you can use to ping the machine with. c.. Status messages appear about once per second per machine in the cluster.

so from the original post it looks like it belongs to a server with an IP address 0A:00:0A:C8 or 10.0.10.200

One snippet that hasn't come up in the thread so far is that the second least significant bit (0x02) of the first octet of a MAC address is the "locally administered bit". That means that any MAC address with that bit set is (supposed) not to be globally assigned but generated locally. Except for a few historical oddities any address with that bit set is not a universally registered OUI.

Sam