IP Cam-Specific Radio Link Problem?

I posted this on comp.os.ms-windows.networking.tcp-ip and somebody suggested taking it here... so here goes...

-----------------------------------------------------------

Just finished 3 weeks of banging my head against the wall.

Windows 7 PC running an IP cam server at location A, radio link to location B where 4 IP cams are running.

Network Diagram:

formatting link

One day, 3 of the 4 cams went down as far as the server was concerned.

The 4th cam, no problem.... ever. Yes, the 3 problem cams are the same make/model (HikVision DS-2CD2032-I) and the "Good" cam is a different make/model (TrendNet TV-IP672P). Both claim to adhere to the same POE standard ("Active 802.3af POE"... whatever that implies) and one of the

3 problem cams is located within 18" of the "Good" cam on with the Cat5e cable the same length.

Here's a screen snap of all 4 cams under Ping -t The "Good" cam being in the center:

formatting link

Go to the remote location, pull the Cat5e cable to the radio link, connect a laptop, start the 4 Ping -t commands, and all 4 cams are running a-ok.... the Replys just keep rolling off the screen.

To cut to the chase, after a loooong time of assuming that no radio link problem could be camera-specific, I re-booted both radios and the problem went away. And, just to put a cap on it, sometime later I was admiring my working cams when all 3 problem cams went offline simultaneously.... again, re-booted the radio links and they immediately came back.

Soooo.... my assumption is that it's the radio link.

The Question:

Can anybody postulate how 3 cams can have a problem and one cam can be running a-ok the whole time?

Or is there a flaw in my logic/assumption?

Reply to
(PeteCresswell)
Loading thread data ...

Active PoE = the power source only applies power when it detects something that will consume it. Passive PoE = power is on the wire at all times.

As to the meat of your problem...tough one. MTU, perhaps? But this is all L2 so maybe it doesn't apply. Buggy networking stack on that model of camera? Is there different firmware available for it [not necessarily newer - you could try a different release on one camera and see how it goes]?

Reply to
alexd

I'll trade you. I burned some quality time last night trying to figure out why a previously working laptop decided to play dead after I tore it apart to replace some broken plastic parts. It took an hour to notice I had forgotten to reinstall the memory chips.

Nice Visio drawing. Do you have a version with all the device details included viewable with IE (active-x required)?

I need a little more detail.

  1. Which camera (by IP or location) is the one that's working?
  2. Are all the devices showing IP addresses setup for a static IP address or are they dynamic? If static, what IP are you using for the default route?
  3. What other devices are on the 10.0.0.xxx LAN that might have a built in DHCP server?
  4. Are the Nanostation M5 radios setup as a bridge or router? If router, do you have the DHCP servers at both ends turned off?
  5. What is the maker and model of the "Extreme West" WAP? If it's a router being used as an access point, is the DHCP server turned off?
  6. What's the maker and model of the PoE switch? Is it a managed switch with an IP address?
  7. What's the maker and model of the Comcast modem/router/wifi box? Some of these are little better than garbage.

Sure. I see it all the time. It's easy to use a wireless router as a bridge or access point. However, things go directly to hell if you have more than one DHCP server running on the system. Make sure every box has the DHCP server turned off, except the Comcast cable modem/router/wireless box. Rebooting the Nanostations to fix the problem suggests that one of them is a possible source of the extra DHCP server.

Dunno. I always ignore the customers diagnostics because they act as a diversion. I look at what you have, what you've done, and draw my own conclusions.

What I see is a mess. Never mind the obvious environmental issues. You have everyone at Toledo Ave and the windsurfing shack all on the same network. You're not going to get anywhere with this unless you can isolate each section and test them individually. Tear it apart, and test each segment individually, starting at the Comcast router. I'm not a big fan of running camera servers on the same Class C IP block as the "productivity" machines, but this is not the time to make the topology more complicated. Still, does the PoE switch support a VLAN?

However, it's always fun to take a wild guess. I'll guess(tm) that if you disconnect the shop PC, shop owners laptops, Extreme West WAP, and any other devices that might have snuck onto the shop LAN, things will magically work as expected.

Reply to
Jeff Liebermann

Something is missing, methinks. It seems odd that the owner of the surf shop would use your wi-fi bridge to connect his store and customers laptops too the internet. More likely, he had a previous internet connection, which is now replaced by the wireless link and your cable modem connection. Any chance that the previous router is still present and plugged in the network somewhere?

The reason I suspect a 2nd router is that I made the same mistake once (and only once). I bridged two locations that previously had impendent connections to the internet. Since the "remote office" then didn't require its own internet connection, I just turned off the power to the router, which was not needed. I left it and the DSL modem connected because I wanted a fall back just in case something went wrong with the wireless link. After a month, the plan was to remove the router permanently.

That worked fine until someone noticed that the router was powered off, and turned it back on. Now, I had two DHCP servers in the network. It might have worked because both DHCP servers were providing IP addresses in the same IP block. However, both routers used the same IP address, and therefore, the same default IP address for the client computers default route, but different MAC addresses. Eventually, ARP requests ended up going to the wrong router and the route to the internet was lost for some (not all) machines. I tried to fix it by remote control, so I couldn't tell that the 2nd router was running. However, I did notice that: arp -a was returning the right IP address, but sometimes the wrong associated MAC address. Even so, it then took a few hours and 2 cups of coffee to figure out the cause.

Anyway, check for a decommissioned router at the surf shop.

Reply to
Jeff Liebermann

Per John Navas:

Installed Wireshark on the server PC and on the remote PC in the shack with the cams.

Now I am climbing the learning curve vis-a-vis WireShark filtering.

Am seeing a lot of 224.0.0.251, which seems to be something called "Multicast"... but not yet able to see the activity from Ping -T

10.0.0.145....

It's going to be awhile.... but that's ok because everything is working right now and I just need to be ready if/when it stops working again...

Reply to
(PeteCresswell)

Per Jeff Liebermann:

No, but that sounds like a feature of Visio that I should look in to - because the drawing is already a bit cluttered. Assuming that those details are behind hyperlinks..

The cam that is working is 10.0.0.133.

All cams have static IP addresses - set both within the cam and with entries in the router's Static IP table.

I do not know what "Default route" is. Will look it up and get back to you.

The camera server PC and the troubleshooting PC at the shack.

The M5's are in bridge mode. On the "NETWORK" tab of each M5 there is a "Management Network Settings | Management IP Address:" which is set to "Static" on both (as opposed to "DHCP") and below that the IP Address, NetMask, and so-forth are all filled in.

The WAP is a D-Link DAP-1522. There is a rocker switch on it with choices "AP", "AUTO", and "BRIDGE". Hopefully it is set to "AP", but I'm not down there so could not swear to it. It does not show up when I run NetScan... FWIW.

El-cheapo, consumer-grade TrendNet TPE-S44. No IP addr, not managed.

Arris TG862G (lotta stuff snipped)

It's easier than that: unplug the M5 at the shop, plug in a laptop, and all is well - as evidenced by a Ping -t command for each cam.

Since it is off-season, the actual network is probably a lot simpler than the diagram implies: just the Comcast box, the cam server, the shop PC, switch, WAP, and the cams.

What I have so far is that one of the prime suspects is some device that is spontaneously going in-and-out of DHCP mode and that the M5's are high on that list.

Next time things go South, I will try to resist the temptation to just remotely reset the M5's and, instead, drive down there and start unplugging things to see what changes/when.

Reply to
(PeteCresswell)

Per (PeteCresswell):

Oops... typo.... sb ...143

Reply to
(PeteCresswell)

I must be slipping. The IP address is all over the screen and I didn't notice: Looks like you have Teamviewer running. Good program and very handy for remote admin.

Ok, TrendNet TV-IP672P parking lot camera. As you mentioned, it's the only one of that type, while the others are HikVision DS-2CD2032-I. I'm familiar with various Trendnet cameras. Fairly straight forward with few surprises or bugs. Make sure you get the updated firmware. Hmmm... do you have a TV-IP672P or TV-IP672P1 ?? They're probably the same as the firmware versions for each are identical.

Checking the emulator: it support UPnP, which has been somewhat of a PITA on IP camera system. If you have port forwarding properly setup, you don't need it. Not much else that I see that might cause a problem. (Yes, I know this camera is working).

The HikVision DS-2CD2032-I camera: has some detail on the network setup in the manual at: Sorry, but I couldn't find anything that might cause a problem.

I still suspect that you have more than one DHCP server active. Find a copy of nmap and run: nmap --script broadcast-dhcp-discover -p67 [IP_address_range] See below for my results. The DHCPOFFER indicates that my DHCP server at 192.168.1.1 is active and working. I'll see if I can find another router later. If there are two, I think you'll see two "pre-scan" responses:

nmap --script broadcast-dhcp-discover -p67 192.168.1.0/24

Starting Nmap 6.46 (

formatting link
) at 2015-01-03 19:14 Pacific Standard Time Pre-scan script results: | broadcast-dhcp-discover: | IP Offered: 192.168.1.165 | DHCP Message Type: DHCPOFFER | Server Identifier: 192.168.1.1 | IP Address Lease Time: 0 days, 0:02:00 | Renewal Time Value: 0 days, 0:01:00 | Rebinding Time Value: 0 days, 0:01:45 | Subnet Mask: 255.255.255.0 | Broadcast Address: 192.168.1.255 | Domain Name Server: 192.168.1.1 |_ Router: 192.168.1.1 etc...

You might also want to use namp to see what happens with the cameras crap out. Just scan for IP addresses that can be seen. If everything on the other side of the wireless WAN bridge goes away, including the bridge IP address, then something is goofy in the bridge. nmap -sP 192.168.1.0/24

Some good examples of using nmap:

Reply to
Jeff Liebermann

Zenmap is similar but has a bare bones gui.

formatting link

Reply to
miso

Per Jeff Liebermann:

Do you recall where you got the icons for the Ubiquiti devices and the modems?

Reply to
(PeteCresswell)

Here's the original Ubiquiti shape archive, which is what I'm using:

This looks newer:

More (with probably some duplicates):

The modem and router are just JPG pictures that I borrowed from somewhere on the internet. Essentially, it's just clip art. There's no device data behind them, although I could easily add that. When I'm in a hurry (or don't care much about the results), I tend to do it this way.

Reply to
Jeff Liebermann

Per Jeff Liebermann:

No chance - I personally dropped the old DSL modem in the trash last time I went down there.

But we keep coming back to that extra DHCP server....

But a PC having a DHCP server is *not* the same as Network Connections | Local Area Connection | Properties | Internet Protocol (TCP/IP) | Properties | Obtain an IP Address Automatically & Obtain DNS server address automatically, right?

That being the case, would somebody have to had install or activate something on the specified PC to make it a DHCP server? Or is it still just a matter of fat-fingering some property?

Also...probably moot... but I just noted this in the server PC's Windows Event Log: "TCP/IP has chosen to restrict the scale factor due to a network condition. This could be related to a problem in a network device and will cause degraded throughput." This is roughly (very roughly.... I have no clue when the problem resurfaced...) coincident with a series of 3 onsets of the original problem today.

Now we are at a place where re-booting the radios makes the problem go away.... but only for awhile (20 minutes? 10 minutes?.... gotta put up a ping log now to get it more exact).

But I note that today it's really wild down there: wind averaging 25, gusting into the low forties.... and the cam views (when they're up) show it with the 2x4 that 2 of the cams are mounted on visibly rocking back-and-forth. Given that the shop radio is mounted on something even whippier (a windsurfer mast) I'm starting to wonder if the radio's swaying around in the wind could have something to do with this.

Bottom line though: can I have a DHCP server active just by fat-fingering a setting, or would something have to have been installed/activated?

Reply to
(PeteCresswell)

Right, they are not the same. The latter case above is an example of a DHCP

*client*.

If it's a Windows server OS, I suppose it might have a DHCP server available, and possibly even running by default, but the desktop OSs aren't likely to have DHCP server capabilities, AFAIK. Do you know what OS is running there?

Routers, OTOH, have a DHCP server on their LAN side that typically runs by default. They also typically have a DHCP client available on their WAN side, so it's possible to have both client and server available at the same time, although not on the same interface.

Reply to
Char Jackson

Per (PeteCresswell):

I think we've struck gold this time.

Posted the above theory in the Ubiquiti forum and two of the resident experts jumped right on me: "You are using shielded cable, right?".....

OOPS!.... RTFM... it's right there in the manual: "ESD attacks are the leading cause for device failures. The diagram below illustrates the areas vulnerable....."

Today's series of failures correlated nicely with peak wind speeds between 10:00 and 16:00:

formatting link

And it's a great time of year to be testing because it gets pretty wild down there on a fairly regular basis.

Three more correlations like this and zero failures otherwise and I'm going to call it "Case Closed".

Now I just have to modify my Ping -T bat file so it knows to start a new text file every time the date rolls over.... then I'll just let it run against one of the problem cameras 24-7....

Reply to
(PeteCresswell)

I've installed 7 pairs of various Ubiquiti wireless bridges and never used shielded cable. Three links are on RF polluted mountain tops and they don't use shielded cable. I also know of other such links that I'm fairly sure (but not certain) that they don't use shielded cable. RF can certainly cause problems, but not disconnect 3 out of 4 cameras, leaving everything else working. That would require some rather unique IP selective RF interference.

Define "failure". Was the Trendnet camera still working?

Wind driven RF interference fixed by adding shielded CAT5? Huh?

For what its worth, most of the problems I've had with links are due to lousy fade margin (weak signals) caused by antenna misalignment, Fresnel zone cancellation, reflections off moving reflectors (cars), and plain old interference. I also had one rediculous problem dealing with mismatched versions (xm versus xw):

I don't gamble, but I'll be willing to speculate that this is going to be an ongoing problem until you identify the cause and culprit. Of course, after you find it, everything will be obvious.

Try FreePing. Set it up to ping literally everything, especially both Ubiquit radios. When something goes down, you'll know what parts of the network are failing. It won't tell you the cause, but it will tell you where to look.

Incidentally, I took a closer look at your ping results at: That's definately NOT what interference looks like. With interference, you get random increases in latency, with ocassional timeout errors. The increases in latency are caused by retransmission delays.

Also, get some data on the signal levels, SNR, and BER between the Ubiquiti radios. The built in graphs aren't good enought for long term monitoring. Offhand, they look ok:

Hmmm... you're on version 5.5.8. Current version is 5.5.10.

I suggest setting up SNMP and running MRTG to make the graphs. If that's too much, setup SNMP and use the demo version of PRTG to draw the graphs. PRTG is easier to setup:

Reply to
Jeff Liebermann

Quite an interesting list of fixes in 5.5.10. Nothing specific to the problem on hand, but certainly some potential problems worth avoiding.

Reply to
Jeff Liebermann

That caught my eye, as well.

Just when I thought that there was nothing new in the world.

Reply to
Char Jackson

Per (PeteCresswell):

Well, I think I have disposed of that little theory by having several failures when the wind was not blowing at any unusual strength.

But I seem to be on the track of *something* with the cam server application. I now have a series of things I can do that will provoke the exact symptoms I have been having.

1) Close the Blue Iris application 2) Stop the Blue Iris service. 3) (observe that cams are still 100% pingable) 4) Re-start the Blue Iris application (which automagically restarts the service) 5) (observe that the three problem cams are no longer continuously pingable) 6) Re-boot both radio links. 7) (observe that all cams are back to normal)

Whhhhaaaaaaat !!!?????

It would appear that somehow the BI application/service may somehow be able to interfere with a camera's continuous pingability AND somehow rebooting the radios remedies that situation.

Jeff: does this seem possible?

Next step is to wait for the homeowner of the server site to come back, provoke the problem, and then have them power off the server PC and see if the pingability returns without rebooting the radios.

Maybe that will tell me whether it's the BI app/service that's stepping on things or some state that it has put the PC into... ??

And if I wind up going down to install the smart switch before they come back, I'll try to install that Belkin remote switch I was talking about before - so I can toggle the PC's power remotely.

Reply to
(PeteCresswell)

Yeah, it's possible. One way is if the Blue Iris server is running a DHCP server and the cameras are accepting IP addresses assigned via DHCP. However, we ruled that out when you said that the cameras were all on static IP's. Still, check for a 2nd DHCP server. Just temorarily turn off the DHCP server in the router. Reboot a few devices and see if they get an IP address. If the get 169.254.xxx.xxx, then there's no hidden DHCP server. However, if they get a valid IP address, then there's a 2nd DHCP server.

Another way to screw things up is to run something like: arp -s 192.168.1.123 00-00-00-00-00-00 where the IP address is some existing device IP, and the MAC address is that of one of the 3 cameras that is failing. In effect, it convinced your client computah that the camera has changed IP address, without actually changing the IP address on the camera. For the grand prize, use the router IP address, and the MAC address of some other device on the LAN, and watch the whole system come to a grinding halt as all the packets destined for the internet to a camera instead of the router. Lots of other variations on the same theme such as duplicated MAC addresses on counterfeit cameras and devices, or Comcast routers that deliver both routeable and private IP addresses to the client machines.

Wireshark and other network sniffers will usually report if there are two IP addresses sharing the same MAC address, or two MAC addresses with the same IP address.

At the risk of being rude, have you tried ANY of the suggestions that I've made in previous postings? I don't think so and you seem intent on discovering the problem yourself. Fine. I'll offer a simple strategy.

Tear the system apart by removing everything that doesn't need to be running, and put it back together section by section until it stops working. Since you now have a way to reproduce the problem, that should be easy. Also, since you're using static IP's, make a spreadsheet listing each devices IP address, Netmask, Default route (gateway), and DNS servers. However, don't take them from what you assume them to be. Login to the management interface, and get the numbers from the devices directly. If ambitious, use some autodiscovery software such as Visio or TheDude: When I have a topology problem, I do it to see if I missed something. That usually finds my sloppy typing errors.

Reply to
Jeff Liebermann

Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.