IP Cam-Specific Radio Link Problem?

Have a question or want to start a discussion? Post it! No Registration Necessary.  Now with pictures!

Threaded View
I posted this on comp.os.ms-windows.networking.tcp-ip
and somebody suggested taking it here... so here goes...
-----------------------------------------------------------

Just finished 3 weeks of banging my head against the wall.

Windows 7 PC running an IP cam server at location A, radio link to
location B where 4 IP cams are running.

Network Diagram: http://tinyurl.com/mkfmvcx

One day, 3 of the 4 cams went down as far as the server was concerned.

The 4th cam, no problem.... ever.   Yes, the 3 problem cams are the same
make/model (HikVision DS-2CD2032-I) and the "Good" cam is a different
make/model (TrendNet TV-IP672P).   Both claim to adhere to the same POE
standard ("Active 802.3af POE"... whatever that implies) and one of the
3 problem cams is located within 18" of the "Good" cam on with the Cat5e
cable the same length.

Here's a screen snap of all 4 cams under Ping -t   The "Good" cam being
in the center:  http://tinyurl.com/ouqrd95

Go to the remote location, pull the Cat5e cable to the radio link,
connect a laptop, start the 4 Ping -t commands, and all 4 cams are
running a-ok.... the Replys just keep rolling off the screen.


To cut to the chase, after a loooong time of assuming that no radio link
problem could be camera-specific, I re-booted both radios and the
problem went away.   And, just to put a cap on it, sometime later I was
admiring my working cams when all 3 problem cams went offline
simultaneously.... again, re-booted the radio links and they immediately
came back.

Soooo.... my assumption is that it's the radio link.


The Question:

Can anybody postulate how 3 cams can have a problem and one cam can be
running a-ok the whole time?    

Or is there a flaw in my logic/assumption?
--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
(PeteCresswell) (for it is he) wrote:

Quoted text here. Click to load it

Active PoE = the power source only applies power when it detects something  
that will consume it.
Passive PoE = power is on the wire at all times.

As to the meat of your problem...tough one. MTU, perhaps? But this is all L2  
so maybe it doesn't apply. Buggy networking stack on that model of camera?  
Is there different firmware available for it [not necessarily newer - you  
could try a different release on one camera and see how it goes]?

--  
 <http://ale.cx/ (AIM:troffasky) (UnSoEsNpEaTm@ale.cx)
 17:25:33 up 68 days,  7:58,  6 users,  load average: 0.28, 0.40, 0.49
 Any sufficiently advanced incompetence is indistinguishable  
 from malice


Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

I'll trade you.  I burned some quality time last night trying to
figure out why a previously working laptop decided to play dead after
I tore it apart to replace some broken plastic parts.  It took an hour
to notice I had forgotten to reinstall the memory chips.

Quoted text here. Click to load it

Nice Visio drawing.  Do you have a version with all the device details
included viewable with IE (active-x required)?

Quoted text here. Click to load it

I need a little more detail.
1.  Which camera (by IP or location) is the one that's working?
2.  Are all the devices showing IP addresses setup for a static IP
address or are they dynamic?  If static, what IP are you using for the
default route?
3.  What other devices are on the 10.0.0.xxx LAN that might have a
built in DHCP server?
4.  Are the Nanostation M5 radios setup as a bridge or router?  If
router, do you have the DHCP servers at both ends turned off?
5.  What is the maker and model of the "Extreme West" WAP?  If it's a
router being used as an access point, is the DHCP server turned off?
6.  What's the maker and model of the PoE switch?  Is it a managed
switch with an IP address?
7.  What's the maker and model of the Comcast modem/router/wifi box?
Some of these are little better than garbage.

Quoted text here. Click to load it

Sure.  I see it all the time.  It's easy to use a wireless router as a
bridge or access point.  However, things go directly to hell if you
have more than one DHCP server running on the system.  Make sure every
box has the DHCP server turned off, except the Comcast cable
modem/router/wireless box.  Rebooting the Nanostations to fix the
problem suggests that one of them is a possible source of the extra
DHCP server.  

Quoted text here. Click to load it

Dunno.  I always ignore the customers diagnostics because they act as
a diversion.  I look at what you have, what you've done, and draw my
own conclusions.

What I see is a mess.  Never mind the obvious environmental issues.
You have everyone at Toledo Ave and the windsurfing shack all on the
same network.  You're not going to get anywhere with this unless you
can isolate each section and test them individually.  Tear it apart,
and test each segment individually, starting at the Comcast router.
I'm not a big fan of running camera servers on the same Class C IP
block as the "productivity" machines, but this is not the time to make
the topology more complicated.  Still, does the PoE switch support a
VLAN?

However, it's always fun to take a wild guess.  I'll guess(tm) that if
you disconnect the shop PC, shop owners laptops, Extreme West WAP, and
any other devices that might have snuck onto the shop LAN, things will
magically work as expected.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
Per Jeff Liebermann:
Quoted text here. Click to load it

No, but that sounds like a feature of Visio that I should look in to -
because the drawing is already a bit cluttered.  Assuming that those
details are behind hyperlinks..


Quoted text here. Click to load it

     The cam that is working is 10.0.0.133.


Quoted text here. Click to load it

     All cams have static IP addresses - set both within the cam
     and with entries in the router's Static IP table.

     I do not know what "Default route" is.    
     Will look it up and get back to you.


Quoted text here. Click to load it

     The camera server PC and the troubleshooting PC
     at the shack.


Quoted text here. Click to load it

      The M5's are in bridge mode.   On the "NETWORK" tab of each
      M5 there is a "Management Network Settings | Management
      IP Address:" which is set to "Static" on both (as opposed  
      to "DHCP") and below that the IP Address, NetMask, and so-forth
      are all filled in.


Quoted text here. Click to load it

     The WAP is a D-Link DAP-1522.  There is a rocker
     switch on it with choices "AP", "AUTO", and "BRIDGE".
     Hopefully it is set to "AP", but I'm not down there
     so could not swear to it.    It does not show up  
     when I run NetScan... FWIW.


Quoted text here. Click to load it

     El-cheapo, consumer-grade TrendNet TPE-S44.   No IP addr, not
     managed.

Quoted text here. Click to load it

     Arris TG862G
      
(lotta stuff snipped)

Quoted text here. Click to load it

It's easier than that: unplug the M5 at the shop, plug in a laptop, and
all is well - as evidenced by a Ping -t command for each cam.

Since it is off-season, the actual network is probably a lot simpler
than the diagram implies: just the Comcast box, the cam server, the shop
PC, switch, WAP, and the cams.

What I have so far is that one of the prime suspects is some device that
is spontaneously going in-and-out of DHCP mode and that the M5's are
high on that list.

Next time things go South, I will try to resist the temptation to just
remotely reset the M5's and, instead, drive down there and start
unplugging things to see what changes/when.

--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
Per (PeteCresswell):
Quoted text here. Click to load it

Oops... typo.... sb ...143
--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

I must be slipping.  The IP address is all over the screen and I
didn't notice:
<https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6093934978803039202
Looks like you have Teamviewer running.  Good program and very handy
for remote admin.

Ok, TrendNet TV-IP672P parking lot camera.  As you mentioned, it's the
only one of that type, while the others are HikVision DS-2CD2032-I.
I'm familiar with various Trendnet cameras.  Fairly straight forward
with few surprises or bugs.  Make sure you get the updated firmware.
Hmmm... do you have a TV-IP672P or TV-IP672P1 ??  They're probably the
same as the firmware versions for each are identical.
<http://www.trendnet.com/support/supportdetail.asp?prod=185_TV-IP672P

Checking the emulator:
<http://www.trendnet.com/emulators/TV-IP672P_V1.0R/mainFrame.xml?nav=Setup
it support UPnP, which has been somewhat of a PITA on IP camera
system.  If you have port forwarding properly setup, you don't need
it.
<http://foscam.us/forum/please-always-disable-upnp-in-your-camera-here-is-why-t7282.html
Not much else that I see that might cause a problem.  (Yes, I know
this camera is working).


The HikVision DS-2CD2032-I camera:
<http://www.hikvision.com/en/us/Products_show.asp?id=9090
has some detail on the network setup in the manual at:
<http://www.hikvision.com/UploadFile/image/2013101808045467171.pdf
Sorry, but I couldn't find anything that might cause a problem.


I still suspect that you have more than one DHCP server active.
Find a copy of nmap and run:
    nmap --script broadcast-dhcp-discover -p67 [IP_address_range]
See below for my results.  The DHCPOFFER indicates that my DHCP server
at 192.168.1.1 is active and working.  I'll see if I can find another
router later.  If there are two, I think you'll see two "pre-scan"
responses:

    nmap --script broadcast-dhcp-discover -p67 192.168.1.0/24

Starting Nmap 6.46 ( http://nmap.org ) at 2015-01-03 19:14 Pacific
Standard Time
Pre-scan script results:
| broadcast-dhcp-discover:
|   IP Offered: 192.168.1.165
|   DHCP Message Type: DHCPOFFER
|   Server Identifier: 192.168.1.1
|   IP Address Lease Time: 0 days, 0:02:00
|   Renewal Time Value: 0 days, 0:01:00
|   Rebinding Time Value: 0 days, 0:01:45
|   Subnet Mask: 255.255.255.0
|   Broadcast Address: 192.168.1.255
|   Domain Name Server: 192.168.1.1
|_  Router: 192.168.1.1
etc...

You might also want to use namp to see what happens with the cameras
crap out.  Just scan for IP addresses that can be seen.  If everything
on the other side of the wireless WAN bridge goes away, including the
bridge IP address, then something is goofy in the bridge.
   nmap -sP 192.168.1.0/24

Some good examples of using nmap:
<http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/


    

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
  
Zenmap is similar but has a bare bones gui.
http://nmap.org/zenmap/


Re: IP Cam-Specific Radio Link Problem?
Per Jeff Liebermann:
Quoted text here. Click to load it

Do you recall where you got the icons for the Ubiquiti devices and the
modems?
--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

Here's the original Ubiquiti shape archive, which is what I'm using:
<http://dl.ubnt.com/UBNT-visio-shapes.zip

This looks newer:
<http://community.ubnt.com/t5/Ubiquiti-Community/Community-Ubiquiti-product-icons-and-stencils-for-diagrams-Visio/ta-p/455771

More (with probably some duplicates):
<http://dl.ubnt.com/media/community/Wiki_Icons.zip
<http://gregsowell.com/?p=1797
<http://wiki.ispsupplies.com/index.php/Ubiquiti_Visio_Stencils
<http://community.spiceworks.com/topic/644325-ubiquiti-unifi-visio-2013-stencil
<https://www.graffletopia.com/stencils/1356

The modem and router are just JPG pictures that I borrowed from
somewhere on the internet.  Essentially, it's just clip art.  There's
no device data behind them, although I could easily add that.  When
I'm in a hurry (or don't care much about the results), I tend to do it
this way.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

Something is missing, methinks.  It seems odd that the owner of the
surf shop would use your wi-fi bridge to connect his store and
customers laptops too the internet.  More likely, he had a previous
internet connection, which is now replaced by the wireless link and
your cable modem connection.  Any chance that the previous router is
still present and plugged in the network somewhere?  

The reason I suspect a 2nd router is that I made the same mistake once
(and only once).  I bridged two locations that previously had
impendent connections to the internet.  Since the "remote office" then
didn't require its own internet connection, I just turned off the
power to the router, which was not needed.  I left it and the DSL
modem connected because I wanted a fall back just in case something
went wrong with the wireless link.  After a month, the plan was to
remove the router permanently.  

That worked fine until someone noticed that the router was powered
off, and turned it back on.  Now, I had two DHCP servers in the
network.  It might have worked because both DHCP servers were
providing IP addresses in the same IP block.  However, both routers
used the same IP address, and therefore, the same default IP address
for the client computers default route, but different MAC addresses.
Eventually, ARP requests ended up going to the wrong router and the
route to the internet was lost for some (not all) machines.  I tried
to fix it by remote control, so I couldn't tell that the 2nd router
was running.  However, I did notice that:
   arp -a
was returning the right IP address, but sometimes the wrong associated
MAC address.  Even so, it then took a few hours and 2 cups of coffee
to figure out the cause.

Anyway, check for a decommissioned router at the surf shop.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
Per Jeff Liebermann:
Quoted text here. Click to load it

No chance - I personally dropped the old DSL modem in the trash last
time I went down there.

But we keep coming back to that extra DHCP server....

But a PC having a DHCP server is *not* the same as  
Network Connections | Local Area Connection | Properties | Internet
Protocol (TCP/IP) | Properties | Obtain an IP Address Automatically &
Obtain DNS server address automatically,   right?

That being the case, would somebody have to had install or activate
something on the specified PC to make it a DHCP server?   Or is it still
just a matter of fat-fingering some property?

Also...probably moot... but I just noted this in the server PC's Windows
Event Log: "TCP/IP has chosen to restrict the scale factor due to a
network condition.  This could be related to a problem in a network
device and will cause  degraded throughput."   This is roughly (very
roughly.... I have no clue when the problem resurfaced...) coincident
with a series of 3 onsets of the original problem today.

Now we are at a place where re-booting the radios makes the problem go
away.... but only for awhile (20 minutes?   10 minutes?.... gotta put up
a ping log now to get it more exact).

But I note that today it's really wild down there: wind averaging 25,
gusting into the low forties.... and the cam views (when they're up)
show it with the 2x4 that 2 of the cams are mounted on visibly rocking
back-and-forth.   Given that the shop radio is mounted on something even
whippier (a windsurfer mast) I'm starting to wonder if the radio's
swaying around in the wind could have something to do with this.


Bottom line though: can I have a DHCP server active just by
fat-fingering a setting, or would something have to have been
installed/activated?

--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?

Quoted text here. Click to load it

Right, they are not the same. The latter case above is an example of a DHCP
*client*.

Quoted text here. Click to load it

If it's a Windows server OS, I suppose it might have a DHCP server
available, and possibly even running by default, but the desktop OSs aren't
likely to have DHCP server capabilities, AFAIK. Do you know what OS is
running there?

Routers, OTOH, have a DHCP server on their LAN side that typically runs by
default. They also typically have a DHCP client available on their WAN side,
so it's possible to have both client and server available at the same time,
although not on the same interface.


Re: IP Cam-Specific Radio Link Problem?
Per (PeteCresswell):
Quoted text here. Click to load it

I think we've struck gold this time.

Posted the above theory in the Ubiquiti forum and two of the resident
experts jumped right on me: "You are using shielded cable, right?".....

OOPS!.... RTFM... it's right there in the manual:  "ESD attacks are the
leading cause for device failures.  The diagram below illustrates the
areas vulnerable....."

Today's series of failures correlated nicely with peak wind speeds
between 10:00 and 16:00: http://tinyurl.com/nb8a846

And it's a great time of year to be testing because it gets pretty wild
down there on a fairly regular basis.

Three more correlations like this and zero failures otherwise and I'm
going to call it "Case Closed".

Now I just have to modify my Ping -T bat file so it knows to start a new
text file every time the date rolls over.... then I'll just let it run
against one of the problem cameras 24-7....
--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

I've installed 7 pairs of various Ubiquiti wireless bridges and never
used shielded cable.  Three links are on RF polluted mountain tops and
they don't use shielded cable.  I also know of other such links that
I'm fairly sure (but not certain) that they don't use shielded cable.
RF can certainly cause problems, but not disconnect 3 out of 4
cameras, leaving everything else working.  That would require some
rather unique IP selective RF interference.

Quoted text here. Click to load it

Define "failure".  Was the Trendnet camera still working?

Quoted text here. Click to load it

Wind driven RF interference fixed by adding shielded CAT5?  Huh?

For what its worth, most of the problems I've had with links are due
to lousy fade margin (weak signals) caused by antenna misalignment,
Fresnel zone cancellation, reflections off moving reflectors (cars),
and plain old interference.  I also had one rediculous problem dealing
with mismatched versions (xm versus xw):
<http://wiki.openwrt.org/toh/ubiquiti/nanostationm5

Quoted text here. Click to load it

I don't gamble, but I'll be willing to speculate that this is going to
be an ongoing problem until you identify the cause and culprit.  Of
course, after you find it, everything will be obvious.

Quoted text here. Click to load it

Try FreePing.  Set it up to ping literally everything, especially both
Ubiquit radios.  When something goes down, you'll know what parts of
the network are failing.  It won't tell you the cause, but it will
tell you where to look.

Incidentally, I took a closer look at your ping results at:
<https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6093934978803039202
That's definately NOT what interference looks like.  With
interference, you get random increases in latency, with ocassional
timeout errors.  The increases in latency are caused by retransmission
delays.  

Also, get some data on the signal levels, SNR, and BER between the
Ubiquiti radios.  The built in graphs aren't good enought for long
term monitoring.  Offhand, they look ok:
<https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6099034450766445106
<https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6099034451179001362

Hmmm... you're on version 5.5.8.  Current version is 5.5.10.
<http://www.ubnt.com/download/#NanoStation:M5

I suggest setting up SNMP and running MRTG to make the graphs.  If
that's too much, setup SNMP and use the demo version of PRTG to draw
the graphs.  PRTG is easier to setup:
<http://wiki.ubnt.com/SNMP_and_MRTG_Monitoring
<http://www.paessler.com/prtg
<http://oss.oetiker.ch/mrtg/

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

Quite an interesting list of fixes in 5.5.10.
<http://dl.ubnt.com/firmwares/XW-fw/v5.5.10/changelog.txt
Nothing specific to the problem on hand, but certainly some potential
problems worth avoiding.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

That caught my eye, as well.

Just when I thought that there was nothing new in the world.

--  

Char Jackson

Re: IP Cam-Specific Radio Link Problem?
Per (PeteCresswell):
Quoted text here. Click to load it

Well, I think I have disposed of that little theory by having several
failures when the wind was not blowing at any unusual strength.

But I seem to be on the track of *something* with the cam server
application.    I now have a series of things I can do that will provoke
the exact symptoms I have been having.

1) Close the Blue Iris application

2) Stop the Blue Iris service.

3) (observe that cams are still 100% pingable)

4) Re-start the Blue Iris application (which
   automagically restarts the service)

5) (observe that the three problem cams are  
    no longer continuously pingable)

6) Re-boot both radio links.

7) (observe that all cams are back to normal)

Whhhhaaaaaaat !!!?????

It would appear that somehow the BI application/service may somehow be
able to interfere with a camera's continuous pingability AND somehow
rebooting the radios remedies that situation.

Jeff: does this seem possible?

Next step is to wait for the homeowner of the server site to come back,
provoke the problem, and then have them power off the server PC and see
if the pingability returns without rebooting the radios.

Maybe that will tell me whether it's the BI app/service that's stepping
on things or some state that it has put the PC into... ??


And if I wind up going down to install the smart switch before they come
back, I'll try to install that Belkin remote switch I was talking about
before - so I can toggle the PC's power remotely.
--  
Pete Cresswell

Re: IP Cam-Specific Radio Link Problem?
wrote:

Quoted text here. Click to load it

Yeah, it's possible.  One way is if the Blue Iris server is running a
DHCP server and the cameras are accepting IP addresses assigned via
DHCP.  However, we ruled that out when you said that the cameras were
all on static IP's.  Still, check for a 2nd DHCP server.  Just
temorarily turn off the DHCP server in the router.  Reboot a few
devices and see if they get an IP address.  If the get
 169.254.xxx.xxx, then there's no hidden DHCP server.  However, if
they get a valid IP address, then there's a 2nd DHCP server.

Another way to screw things up is to run something like:
  arp -s 192.168.1.123 00-00-00-00-00-00
where the IP address is some existing device IP, and the MAC address
is that of one of the 3 cameras that is failing.  In effect, it
convinced your client computah that the camera has changed IP address,
without actually changing the IP address on the camera.  For the grand
prize, use the router IP address, and the MAC address of some other
device on the LAN, and watch the whole system come to a grinding halt
as all the packets destined for the internet to a camera instead of
the router.  Lots of other variations on the same theme such as
duplicated MAC addresses on counterfeit cameras and devices, or
Comcast routers that deliver both routeable and private IP addresses
to the client machines.

Wireshark and other network sniffers will usually report if there are
two IP addresses sharing the same MAC address, or two MAC addresses
with the same IP address.

At the risk of being rude, have you tried ANY of the suggestions that
I've made in previous postings?  I don't think so and you seem intent
on discovering the problem yourself.  Fine.  I'll offer a simple
strategy.  

Tear the system apart by removing everything that doesn't need to be
running, and put it back together section by section until it stops
working.  Since you now have a way to reproduce the problem, that
should be easy.  Also, since you're using static IP's, make a
spreadsheet listing each devices IP address, Netmask, Default route
(gateway), and DNS servers.  However, don't take them from what you
assume them to be.  Login to the management interface, and get the
numbers from the devices directly.  If ambitious, use some
autodiscovery software such as Visio or TheDude:
<https://www.mikrotik.com/thedude.php
When I have a topology problem, I do it to see if I missed something.
That usually finds my sloppy typing errors.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: IP Cam-Specific Radio Link Problem?
Per John Navas:
Quoted text here. Click to load it

Installed Wireshark on the server PC and on the remote PC in the shack
with the cams.

Now I am climbing the learning curve vis-a-vis WireShark filtering.

Am seeing a lot of 224.0.0.251, which seems to be something called
"Multicast"... but not yet able to see the activity from Ping -T
10.0.0.145....

It's going to be awhile.... but that's ok because everything is working
right now and I just need to be ready if/when it stops working again...
--  
Pete Cresswell

Site Timeline