VOIP Degradation on Wireless Network

I may be leading you up the garden path here. There are two PCs in my house connected to the network - one is a work PC and one is the home PC and they are running mostly different software. I have focussed most of this troubleshooting on the work PC because that's the one I want to use for IP telephony. The other one I checked briefly and got the impression it had the same issue, but I was too hasty in my checks late the other night.

Last night I checked again - the other (home) PC has a different sequence and pattern of latency spikes than the office one. I put them right next to each other and the pattern which happens ever minute or so on the office PC does not happen on the home one, at least not at the same time. The home one actually has more frequent issues - some of them are characterised by just one long ping or lost packet, and others are characterised by a pattern just like the one you looked at yesterday, but not on the same timing - they are more frequent than once per minute and less regular.

So that seems to indicate that it's something to do with the local PC and software running on the PC that's causing the issues (?), or extremely localised RF interference caused within the PC itself (?). I have tried disabling everything in the startup file using MSConfig but that didn't make any difference. Are there any services that you have seen that could be culprits here?

BUT when I connect to the router with an ethernet cable, I get no latency spikes whatsoever, so if it's something running on the local PC it is something that only impacts on the wireless networking. If something was stopping the network or grabbing all the bandwidth whilst it phones home, I would have thought I would see at least some impact on wired networking as well.

Thinking whilst I am talking though, I seem to remember reading that on a wireless network, only one device can actually communicate at a time in the network layer (?) and the communication can only go one way at a time. Therefore could it be that the home PC is grabbing all the network bandwidth every minute and the other one doesn't get a look in? I don't think so though, because I have also tried tests with the home PC completely switched off (and everything else in the house switched off as well) and get the same results.

I'm a bit stumped now, but local software now seems to be the most likely culprit?

Sorry for the misleading information before - it must have been too late at night last time I checked the home PC :-)

thanks Patrick.

Everything in the list above that I have in my house has been disabled

- I even took the batteries out of a wireless mouse and keyboard upstairs. I did not take the batteries out of all the kids toys, but as far as I can remember they don't have any radio controlled toys. To my knowledge the burglar alarm is wired not wireless, and I would not have a clue how to switch it off without triggering a tamper alarm. It is not the type of alarm that alerts the authorities, so I doubt that it has radio transmission capabilities. The motion detectors have wires that disappear into the walls, so they are not wireless.

I don't know what a Zigbee is so I couldn't comment on that.

Also see my other post where I had actually misdiagnosed the other PC in my house, and the interference patterns are NOT the same on both PCs.

Other comments:

- I tried using a different router (US Robotics 9106) that I had stored away and the results on both PCs were the same as with the BT router.

- If I move the office PC which is a laptop around the house, there are places where I seem to get very similar interference patterns much more frequently, but even depending on whether there are objects blocking the way and/or which direction I am facing), but the underlying ding every 60 to 70 seconds always seems to be there anyway.

- Of course I guess in theory any of the items mentioned above (and any others on the list of frequent causes of RF interference that was linked in one of the messages above) could also exist at the neighbours house - my house is a terrace so it's joined to the neighbours on both sides.

cheers and thanks for all your help on this. We may not find the issue but I'm certainly more clued up about wireless networking issues than before you have been extremely helpful.

warner snipped-for-privacy@hotmail.com hath wroth:

I'll assume that both PC's are connected via wireless. Putting them next to each other is no guarantee that you'll get identical wireless performance and interference. However, I've been assuming that the point of entry of whatever is causeing the interference is the cental wireless access point, and not the client radios which are usually less sensitive with inferior antennas. That may be a bad assumption.

Note the extra packet loss at about 750msec before the 1 second long loss. I suspect that you're watching a transmission between TWO other wireless devices, a weak one and a strong one. That will make the actual pattern somewhat variable depending on location.

That's easy to test. Move the PC around. If it's software, shouldn't mattery where the PC is located, the level of interference and interference timing pattern should remain roughly identical in all locations. If it varies in intensity and possibly pattern, it's something external. If it stays the same at all locations, it's something in software.

Again, easy to test. If it's coming from inside, it will remain the same no matter where you're located. Run the XP Task Manager Performance graph and see if the CPU gets very busy or spikes at corresponding 1 minute intervals.

Yes. I think I previously mentioned a few. It's any service or background application that regularly checks for network connectivity. Skype and all VoIP applications do that. Any network monitoring programs that use ping or SNMP. However, I really doubt that these are the problem due to the long duration of the interference. 1 full second is an awful long time for a software application to hog the CPU. If it had been a momentary (I don't think so though, because I have also tried tests with the home

Good test assuming the transmissions are coming from your equipment. If it's coming from the neighbors, it might be a problem.

No, I don't think so. I think it's time to do a bit of isolation testing in order to reduce the number of possibilities. Disconnect your BT wireless router and go find a location where little chance of RF interference. I suggest a basement, neighbors house, dungeon, or similar isolation chamber. You won't need internet connectivity for this test. Take your laptop with you. Run the fping 250msec test and see if you get the same 1 minute interval, 1 second outage. If it travelled with you, then it's something either inside the BT router or the laptop. If you left your interference problems at home, then it's something at or near your location and most likely a source of RF interference.

Well, actually the most important information that was missing was the available hardware and topology (method of connecting to the network).

Some questions:

1. Is your home PC connected via wireless?
2. Can you see any neighbors wireless networks on the laptop with the "View available networks" or other tool (Wi-Fi Hopper, Netstumbler)?
3. Have you tried taking the laptop yet a different location such as a wireless hot spot and tried the Cisco VoIP?
4. Got any friends with 2.4GHz spectrum analyzers?

warner snipped-for-privacy@hotmail.com hath wroth:

I'll assume it's not coming from your house. That leaves the neighbors and the neighborhood.

It's possible that the 1 minute interval, 1 second burst, is wide band noise from something that's arcing. It should be possible to hear it on other radio equipment. If you have a portable AM/FM/TV device, try wandering around and see if you can hear or see something while listening to an empty channel.

Incidentally, one of the local hams was trying to locate the source of a similar RF interference pattern that was ruining his HF (short wave) communications. It was eventually traced down to some manner of pump controller for the apartment building. I think it was the swimming pool pump but may have been something else. The noise came from an arcing thermostat. The pattern you observed seems similar. Got any mechanical thermostats driving high current loads?

Smoke alarm? Mine has a battery saver circuit that samples at about 1 minute intervals. It probably takes 1 second to make a measurement. Hmmmm....

I'll make it easy. Are you next to a university or industrial plant? They use Zigbee radios for mesh wireless sensor networks.

I would't expect them to be identical. I think you're hearing someone elses RF transmissions. Exactly what and from where, I don't know. Since changing the channel doesn't work, I suspect a frequency hopper and not an 802.11 direct sequence radio.

It might also be one of the neighbor un-friendly MIMO or turbo devices, but the inteference pattern doesn't fit to well. Well, actually it might fit under an unusual case. If the neighbor has one of these devices, but is not using it for moving any traffic, the access point will switch to a "sampling" mode spending the bulk of it's time doing 802.11g, and ocassionally listening for turbo or MIMO transmissions. I'm only familiar with the transmission patterns from a few of these (Airgo based systems) and have noted that it changes with firmware, model, manufacturer, and technology. I'll do some digging and see if I can find a pattern of what it looks like at idle. I don't think this is the cause because at the level of interference that it seems to produce, you should have been complaining about much longer duration interference when the MIMO system was in use. Still, it's worth persuing.

Ok, that eliminates the BT router as a possible culprit. Just for fun, turn off the cable or DSL modem. You don't need it for the ping test.

That seems to show that it's RF interference. This is going to be difficult to identify without test equipment or doing something drastic like powering down the entire house. Might as well. Power the access point from a UPS and turn off the entire house.

Well, ask the neighbors if they have anything on the list of possible interfence sources. Bring your laptop when you visit to see if it gets worse or better at the neighbors. My guess(tm) is that 30m in any direction is as far as you need to go.

I find this kind of troubleshooting to be better than any mystery novel. Good luck.

Unless there is more than one issue, and one is internal one external. Thanks for your usual very thorough reply - I have some family commitments this weekend so I probably won't be able to action the rest of it until Monday or so - but I will review all that for sure.

cheers Patrick.

OK - I did have a bit more time to investigate this evening:

Yes.

No

No, but that's a very good idea and I may try to do that soon. That will probably be the final most effective test to see if it's something in the laptop or something external in the area of the housr.

Un, I'm not even sure if I have any friends who would know what a

2.4GHz spectrum analyzer is. Are they cheap to buy? Before I go down that route I want to totally disprove that it's not something on the PC causing it.

Lastly, some new evidence which I'm not sure if it's cause or effect - I just realised that there is a very clear spike in CPU usage on the laptop that corresponds to the network outages - it's not a spike to

100% - it's a spike from 1% to about 12%, but it clearly is happening bang on every time the fping outage occurs. This spike only occurs when wireless networking connection is active. It does not occur if there is no network at all, and it does not occur if there is a wired connection. It is not caused by FPING as it happens even when fping is not running if the wireless network is active. Is it normal to see a CPU spike when there is an RF interference incident on the wireless network?

cheers

MetaGeek Wi-Spy 2.4 GHz Spectrum Analyzer $100 Bumblebee $3,000 (depends on model) and you need an iPaq Cognito and Airmagnet PCMIA cards $3,000 - $4,000 Stand alone units $10,000+

As Mr. Decaturtxcowboy mentioned, Wi-Spy for $100 is a bargain. The problem is that it's not very sensitive and not at all directional. I sorta solved the directional problem with a salad bowl antenna: |

| are other forms of kitchen reflectors: | If you've never used a spectrum analyzer before, it will be difficult to know what to look for. There are sample capture files on the Wi-Spy web pile: | suspect you're looking for something fairly weak and obscure which may be difficult to recognize. Probably the best approach would be to convince the company IT department that it really needs to buy one, and then borrow it. However, wait for the new version that has an SMA connector for an external antenna. The salad bowl is really a bad way to play direction finder.

There are plenty of others available at exponentially increasing prices and complexity.

Hmmm... Good observation. No, it's not normal. Wi-Fi network interrupt overhead is minimal (unless you have a seriously underpowered laptop). Is there anything that might make the CPU work harder such as lack of adequate RAM or very slow CPU?

If the outages really do correspond to the CPU usage spikes, then the problem is in the laptop. However, if it's in the laptop, it would also have appeared at the office as well as home. I think you can identify the culprit program or service using the Task Manager. Instead of the Performance Graph, select "Processes". Click on the top of the "CPU" column once or twice to get it to sort on CPU usage. The process that eats the most CPU cycles will appear at the top of the list.

You might also want to add "CPU Time" to the display: Task Manager -> View -> Select Columns

Drivel: Try the same test for interference with the laptop battery charger disconnected. It just might be junk from the Li-Ion battery charger circuit. If you use a different power supply at work, I would definately look into this one.

decaturtxcowboy hath wroth:

Avcom $2700. |

Bantam Instruments $2950. | Using an MMDS downconverter, a much cheapter 500Mhz or 1GHz spectrum analyzer can be used.

There are also the Proxim FHSS 7400 RangeLAN2 based spectrum analyzer cards for about $20. They sorta work but not for intermittent sources of interference. The sweep rate is just too slow: |

If you don't mind used, there are quite a few good spectrum analyzers on eBay:

HP 8569B for $2700. |

and slobber... I want it, I want it, I want it. Argh.

">

Not that I'm aware of and the CPU was flat at 0% for all the intervening time. It's a Dell Latitude D600 - not state of the art, but it should cope with a few networking tasks fine.

I already tried that, but I didn't add CPU time in - I tried just sorting by CPU to see if any programs were spiking on CPU usage at that moment but I couldn't identify any. I also tried closing down the programs starting with the busiest, until I ran into accidently stopping an essential process and the machine shut down. I couldn't identify a culprit yet in this way. I'll try it with CPU time also. Of course, if the culprit is actually the wireless network drivers or configuration utility then I'm stuffed - to check for that I will need to try a different network card - actually I just rememered I think I have a spare PCMCIA wireless card around somewhere so I might try that.

In the office I connect wired not wireless. I am going to our US office next month where there is wireless, so I can see what happens then, or as you said I could also take it to a hotspot. But again it's interesting that these CPU spikes don't happen when I'm connected wired, only wireless. I am pretty confident about this because I sat and watched through a 5 minute period each with wireless/wired/no network. With wireless, there is a clear CPU spike that corresponds to when I get the network dropouts. (at this point many others are probably thinking I should get a life instead of spending Friday evenings watching the task manager, but there you go).

I'll try that too.

"Patrick" hath wroth:

Try increasing the sample rate in Task Manager. View -> Update Speed -> High

The Dell D600 usually arrives with a Truemobile 1300 MiniPCI card using BCM4301 chipset. Is that what you have? It's a generally acceptable card with no unusual problems that I could find.

Duh... Try turning off the wireless using the Fn key method and the right click on the System tray icon, and see if the 1 minute glitches go away. If it goes away, the glitches are definately coming from the wireless card or driver.

The MiniPCI wireless cards are rather dumb. Much of the wireless dirty work is done by the main CPU. I have customers with Dell D600 or D610 laptops and can check if they have the same one minute busy cycles. It's possible that it would be a common trait as it would only cause a problem with realtime applications like VoIP.

Anyone else have a Dell D600 handy that can check?

Ask yourself "What did I do on Friday evenings before discovering computers"? If you can't remember, you have a problem.

Jeff Liebermann hath wroth:

It may also help to get a better Task Managerie. See Sysinternals (now owned by Microsoft) Process Explorer 10.21: |

can run at 0.5 sec per sample, which should be sufficient to see a

1 second long hickup. Click the mouse on the CPU column to get the processes that are active at the top of the screen.

Right, let's recap where we are:

1) Over the weekend I took the laptop to a hotspot and I found the same symptoms when logged into the hotspot. That means the problem is coming from the laptop or from something upon my person - I had already eliminated my cellphone from enquiries, so the only other electronic object on my person is my watch - I doubt it's that.

2) Disabling radio on the task manager - I think I already had effectively tried this in a previous message - last week I looked for the CPU glitches whilst there was radio disabled and no network cable connected. The glitches were not there. They only occur when I am connected to a wireless network. In fact, if radio is enabled, but I am not connected to a network, they don't occur either - only when I am actually logged on to a wireless network. Therefore this seems to indicate that it's something specific to the wireless card drivers or utility, or at least a conflict between that and something else.

3) My Wireless card is listed as a "Dell Truemobile 1400 Dual Band WLAN Mini-PCI Card". The manufacturer is Broadcom. I don't know what the chipset is. I did also notice just now that there are a whole bunch of settings that can be changed from within the Device manager for the network card - I never tried changing any of them.

4) I still can't isolate anything using process manager even when set to high refresh rate, but I will download and try that other process manager you mentioned.

5) I need to check the Dell website for updated drivers - I checked it several months ago and there was no update for a couple of years, but I'll check again.

6) Battery charger, I have tried it without the battery being on charge and no change. Should I also try it with mains power but no battery in the PC?

We are getting closer.....

cheers Patrick.

1. Bluetooth on that laptop? If so, disable it temporarily.

1. Run thorough anti-virus and anti-spyware scans.

2. Use MSCONFIG to temporarily disable background processes.

On 13 Nov 2006 14:02:10 -0800, warner snipped-for-privacy@hotmail.com wrote in :

Well, that eliminates interference and the BT access point as possible culprits. It's something in your laptop or wireless device.

Weird. The only think I can think of in the driver that ocurrs at 1 minute intervals is the "power save" feature. Look for it in the properties for the wireless device. I suggest you disable the power save mode.

It might also be a program or service that's checking for internet access. If it finds internet access, it monopolizes the processor for about a second. If it doesn't fine internet access, it doesn't do anything, which is why it doesn't show up every minute. However, that's unlikely as it should act the same with wireless and wired connections. I'm beginning to agree that it's something in the wireless card or wireless driver.

These are the same setting as found in the properties for the wireless device under "Network settings". Try to remember the default settings. I don't have any experience with this particular card. The Truemobile 1300 (2.4GHz only) is a fairly common device, but not the

1400. If you feel like burning a few dollars, you might try a 1300 PCI from eBay for about 37 GBP.

I'm not sure it will be much better, but it does show quite a bit more detail and is somewhat faster.

There appear to be some recent Truemobile 1400 drivers. Be sure to grab the correct version for UK. |

No, don't try it without the battery. The idea was to see if the battery charger "pulsing" with a fully charged battery was causing the

1 minute glitches. It was a long shot.... oh well.

I hope so. I like to fix things like this by substitution. If you can scrounge another wireless card, try it instead. Just make sure you create a system restore point before attacking so you can put things back to current condition.

On Tue, 14 Nov 2006 02:45:54 GMT, Jeff Liebermann wrote in :

Worth trying, but I doubt that's the cause. I suspect something else running in the system, even given those symptoms. I'd personally use MSCONFIG to disable everything possible, in addition to running malware scans.

Yep. Lots of shitty software out there. Processor spin loops are lame but all too common.

Not necessarily -- it could easily be due to differences between wired and wireless networks -- as you've noted, wireless cards typically depend on much more host processing.

Will try that later.

OK Jeff - I have loaded up an old US Robotics 5410 PCMCIA wireless card and disabled the on board card.

Results are quite positive.

Firstly, the one minute glitches have disappeared, so we have isolated those 1 minute glitches to something to do with the on board wireless function.

Secondly, the real world voice over IP performance seems much better - I am geeting a good connection with virtually no interference.

On the minus side, my bandwidth seems to have taken a nosedive with the US Robotics card - I'm down to about 2Meg compared to 6 Meg with a wired connection. Also, when I run FPING, I am actually seeing more dropouts than with the on board card - I am seeing long pings and timeouts probably every 20 secs or so, but not totally regular. However as I say this doesn't appear to cause a break in IP communication. I think maybe the difference is that only one packet is dropping instead of a continuous issue that lasts for a whole second.

However if I can get good IP telephony I may be happy to go with that, even if my bandwidth seems to be restricted by this USR card. I seem to remember someone telling me the USR wireless cards were not very good? Do you have any knowledge on that?

thanks Patrick.

warner snipped-for-privacy@hotmail.com hath wroth:

That's an 802.11g card and should work (famous last assumptions). It's not that old as USR lists it on their web pile. |

Agreed. That generally eliminates any programs that are not directly related to the Dell TrueMobile 1400 card and driver. We still don't know if it's the card or the driver. That's why I suggested you buy or borrow a 1300 card. I forgot to check with my customers that might have a D600 or D610. Sorry.

Ah... progress.

That sounds like local interference. The drop in speed could be due to massive retransmissions. Not good. Are you SURE you have the internal card turned off? There are usually 3 ways to disarm the card. One is in the CMOS setup which disables the MiniPCI slot for reduced power consumption. Not every BIOS has this feature. Another is on the keyboard, which I think (not sure) also disables the power to the card. Last is in Windoze from the system tray. This just disables the NDIS5 driver and may not totally turn off the card. It's easy enough to literally extract the card. If you suspect that it's still running, try a Wirelessectomy.

Possible, but difficult to tell with all that packet loss.

No, not me. I have some experience with the ancient Eumitcom based wireless products from about 1998. I have a few of their access points from this era sitting in the closet doing nothing. They were not very good at doing anything, but then neither was the competition at the time. No experience with their PCMCIA cards.

I'll take another look at that - and see if I can disable wireless in the CMOS - I disabled it by disabling the wireless device in the device manager.

cheers Patrick.