Lighning hit a 12kv line very near. (about 30-40m away). This was downstream from the transformer that feeds me electricity and there was a fuse between the point of impact and our transformer.
A mac with long ethernet cable had its ethernet interface zapped. But otherwise the mac still works.
It was plugged into port 10 of a Cisco 2924XL switch.
The switch rebooted, but amber lights remained on ports 9, 10, 11 and 12.
9 and 11 were local devices (printer and one mac)
10 was the distant imac. All these 3 were turned off at the time of the event. These 4 ports are inoperative.
The other ports seem to function. But the main status light remains amber.
is it correct to assume that the hardware drives groups of 4 ports so if one port goes bezerk, it brings down the other 3 in that group ?
Out of curiosity, what IOS command would tell me about the error in those ports ?
show interface doesn't give anything of use: SWITCH1#show int fa0/10 FastEthernet0/10 is down, line protocol is down Hardware is Fast Ethernet, address is 0004.4dfd.1a8a (bia 0004.4dfd.1a8a) Description: IMAC MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive not set Auto-duplex , Auto Speed , 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 02:40:49, output hang never Last clearing of "show interface" counters never Queueing strategy: fifo Output queue 0/40, 0 drops; input queue 0/75, 0 drops 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out SWITCH1#
What commands would show me the errors on the switch ? (I currently can't look at the serial port while the switch is booting because my cluster freezes during that time).
Would there be a way to disable those 4 ports so that the unit would then report a healthy status ?
Also, out of curiosity, would nearby lightning induce currents on long ethernet cables which would have explained this ? (devices near the switch in the basement were not damaged).
Lucky. :) You can never predict what a surge will do.
Yep I have lots of experience with exactly that failure. On the 2924xl, the PHY's are in blocks of 4 chips, same on some of the newer lines. Its probably a blown PHY chip. But they are surface mount quad-flat pack chips with over a hundred pins. Ie. a surface-mount rework machine is going to be needed to swap the chip.
During a reboot, you'll probably get the switch complaining about those 4 ports not passing POST in the log. Sometimes not.
Once you start doing hardware frying, its not going to be too easy to report errors on it, when the hardware doesn't behave like you need it to.
No, you'll need to get it repaired or replaced. Smartnet if you can still get it on this long EOL'd switch was fairly cheap, $125?
Yep, sure does. Especially the closer ethernet runs are to the outside. I had a building once I supported that ran all the ethernet under the roofline. Routinely blows a switch every storm. We had input as to where the wire should go, but they didn't listen, just went with the cheapest wiring contract they coul. Guess that decision was a mistake :) At least it wasn't ours.
If you ever trench ethernet outside, this is exactly why you'd want fiber in the ground, instead of copper.
Well, since the switch has amber lights on the block of 4 ports, and since its own main status light is also amber, I would have thought the software would know about the problem. (and would think that some error message would appear when you do an show interface fa0/xx for one of the faulty ports.
Or buy a "new" one from ebay. But I'll call Cisco on monday to find out.
Thanks. That was probably it then. I guess I can consider ourselves lucky we didn't get a direct hit. The neighbours in the back whose house is just a few metres from the pole that was hit are probably feeling even luckier.
Half the townhouse complex is still without power. They not only have to replace their underground transformer, but also replace the underground cable that goes from thetransformer to that pole. (it is that cable that was zappep and Hydro guys tols me tha don't splice this type of cable.
I hadn't thought about EMI for thunderstorms for that, always tought about grounding differences for this issue.
Back when we were doing RS232 and X.25 we had copper runs in one particular place that used to get lightning strikes. The X.25 PAD used to suffer from, at best, the tops blown off the RS232 driver chip packages and at worst holes through the board, like a bullet hole, where the chip used to be.
Your description is typical of a "direct hit". How will be explained later. From your description, surge electricity flowed through port 10 and through the connected computer. Remember, to have electricity; first an incoming and outgoing path must exist. Which side was the path to earth? Makes no difference whether electronics were powered on or off.
In another example, a plug-in protector earthed a surge through an adjacent computer, out via network wire, into network card of another powered off computer, out via that computer's modem, to earth ground via phone line. We traced that surge by identifying and replacing every damaged semiconductor.
Possible that four ports in the Cisco share a common interface chip. ICs for such ports also come in four or eight port versions. This is speculation that would explain why surge current through port
10 would damage ports 9 through 12. Dead body would better explain why damage occurred.
This we do know. It was a direct strike. A second example. A nearby tree was struck. Campers sleeping tangent to that tree did not suffer. But two campers sleeping pointed to that tree suffered a direct strike. Electricity flowed down the tree into earth. Up into camper's feet and back to earth via heads. Current flows some miles distant to electrical charges. That was a direct strike to campers whose body provided a better electrical path.
Ethernet (communication) connections between buildings so easily suffer damage for same reasons. Third example: one building literally becomes the lightning rod (or tree) to conduct a surge through ethernet cable to be earthed in other building. That path to earth is destructive to electronics - powered on or off - because it easily overwhelms protection already inside ethernet interfaces.
Many then assume damage is from induced surges - or EMI. Numbers say otherwise. The concept is called GPR. Solution requires single point earthing of every incoming wire, short, to a single point ground. Any wire of any cable not earthed, short, by hardwire or protector may carry destructive surges inside the building. That other building simply acts like a lightning rod to carry surges into this building. Damage for reasons similar to those campers.
Also at risk are four legged animals - for same reasons - a fourth example. Properly constructed barns install an even better single point ground - halo or Ufer ground. Single point earthing that encircles a barn means no current will flow up a cows hind legs and down its fore legs. This direct strike (when lightning strikes a nearby tree) is reason for livestock deaths. Learn from that example to install that inexpensive and effective earthing in new buildings. However, even that earthing is useless when incoming wires are not earthed before entering a barn or network center.
Back to the original failure. What was the incoming path of the surge that was seeking earth ground? Autopsy of 'dead bodies' would be useful. But finding someone with sufficient electrical knowledge is difficult.
What we do know: surge found one earthing path via Cisco port 10 and a computer's ethernet interface. Surge could have been incoming on AC electric through computer, through Cisco port 10, then ... well, could ports 9, 11, or 12 conduct to earth? What thing conductive was the Cisco mounted on? Even furniture could have been part of an earthing path.
We know a surge entered the building on a wire that was not 'single point' earthed via hardware or protector. Fields do not provide sufficient energy to overwhelm a many thousand volt protection internal in computer and Cisco. How many thousands? Well, IEC standards now define interface chips that must withstand 2K or 15K volts. Again, we don't know which were inside those 'dead bodies'. But we know EMI will not overwhelm that existing and industry standard protection. It was a direct strike - if only just like those campers beneath a tree.
How to prevent such damage? Well your telco have been connected to virtually every building in town. Every building is a lightning rod that connects surges directly into their switching computer? But telco installed protection has been standard everywhere since the beginning of the 20th Century. Preventing damage has been that well understood for that long. Each incoming wire must make a short connection to a common earthing electrode. Each wire connects via a 'whole house' type protector so that damage will never happen. Your damage implies that well proven and properly earthed protection does not exist in your facilities.
Little difference whether wires are underground or that lightning struck 30 meters distant. Fifth example: an application note from Polyphaser:
formatting link
Lightning strikes somewhere across the street close
Those quick to speculate may instead blame damage on EMI. Too many examples and those numbers are little understood by those who blame EMI. It was a direct strike that overhwhelmed protection found on all ethernet ports. Power on or off makes no difference. Solution starts with what always defines protection - the earthing system.
Thanks. Unfortunatly, they don't seem to show any errors. SHOW TECH produces a list of the "SHOW INTERFACE fa0/xx" commands (which show no errors/problems for those 4 ports), yet amber lights are on for those 4 ports, as well as one of the main status lights on the left of the unit which is amber instead of green.
I am expecting a replacement unit in a couple of days and will then be able to do more experiements in this damaged switch. (especially look inside to see if there are traces of the smoke I smelled after lightning struck).
Most all surges leave no visible damage. Visible damage is usually due to something called 'follow through current'. IOW a surge creates this tiny little damage. A resulting short circuit permits massive energy from AC mains to do further damage - create smoke.
Recording IC part numbers and manufacturer logo for ethernet interface chips would also answer more questions such as how many ports are controlled by one IC. Also part of a circuit (why ethernet ports are so robust) would be a transformer between each ethernet wire and that IC. Electrical characteristics of that transformer (does galvanic isolation still exist?) may provide further insight.
If not obvious, autopsy is how experience educates from the 'experimental' side of knowledge. Useful experience comes from repairing things at the component level. Then mate that with 'theoretical' concepts (the other side of knowledge) detailed in IEEE papers, et al. Reason for doing so is to identify where lightning entered the facility. Only then is defective earthing identified (with confidence) and repaired.
SWITCH1#show post POST FAILED: FastEthernet0/9 failed front-end loopback test POST FAILED: FastEthernet0/10 failed front-end loopback test POST FAILED: FastEthernet0/11 failed front-end loopback test POST FAILED: FastEthernet0/12 failed front-end loopback test SWITCH1#
So that was the magic command to show the errors.
Replacement switch coming in today, so I should be able to then do a good post mortem on the damaged switch !
Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.