Rebooting radio in Santa Cruz mountains once a week via GitHub script?

Have a question or want to start a discussion? Post it! No Registration Necessary.  Now with pictures!

Threaded View
If we wanted to automatically reboot our radios once a week, should we do
it with a GitHub script on the radio or do you know of a better way?
<https://github.com/stevejenkins/unifi-linux-utils

We're on a classic small WISP outfit (2-man operation, maybe 100 homes) in
the Santa Cruz mountains, where, whenever we call for technical support,
they *always* tell us to reboot the radio - where - shockingly - often -
the performance goes up (don't ask me why or how - it just does).

So let's not argue the fact that our WISP Internet speeds are better (for
whatever reason) after we reboot our rooftop radios (which most of you
would call a "modem" but we call them transceivers, or just radios, for
short).  

Let's just ask the question.

If we wanted to automatically reboot our radios once a week, can we do it
with a GitHub script on the radio or do you know of a better way?

For example, how does this GitHub script look to you Linux coders like
Marek Novotny and William Unruh?
<https://github.com/stevejenkins/unifi-linux-utils

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On 11/14/17 21:19, harry newton wrote:
Quoted text here. Click to load it

Hope you really have another admin user/password than ubnt.

Quoted text here. Click to load it

If you want the reboot to occur periodically, then a crontab job would
be simplest.

If you want more control, then have periodical checks of a site, and if
the value changes on the requested page say from 0 to 1, then reboot
(then have a grace period when it will not check the page, or else it
will keep rebooting until you change the value back to 0).

If making the page with some kind of logic, then each client sends a
hash unique for them to you, if the value is in your database, then send
value 0, if not send 1 and each time you want all units to reboot you
empty the table.

Sure you could use nrpe to do the same thing as your ssh connection, but
with the difference that you wouldn't need to know the password of the
device, lock it with ip restriction. This will work quite nicely till
someone does an arp poisoning and make it look for the device that you
would be sending the nrpe request while it's someone else.
I have used nrpe where the client been allowing request from a dynamical
ip server, this has the small drawback that you have a minute while you
can't access the nrpe service, but I do rather use it than ssh which
requires you to configure username/password in nagios/icinga.

--  

 //Aho

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Tue, 14 Nov 2017 20:19:38 +0000 (UTC), harry newton

Quoted text here. Click to load it

I use a AC lamp timer to power cycle everything:
<https://www.intermatic.com/en/timer-controls
Pick your flavor, they all work.  I've been using these:
<https://www.intermatic.com/en/timer-controls/plug-in-timers/dt620mx
The main problem is that the tiny button batteries don't last long
enough.

The problem is that I don't trust software solutions to fix a problem
caused by an operating system or application that has gone insane. The
reset has to be done by something that won't be trashed, hung, or
become too busy, which means an external device or independent
"heartbeat" timer.

Before you complain about crashing your system by killing the power,
please note that if your stuff can't handle a power glitch or failure,
it doesn't deserve to be run unattended.



--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Tue, 14 Nov 2017 21:17:02 -0800, in

Quoted text here. Click to load it

Klugey approach?

Set a WatchDog Timer reboot.  

Specify an IP that won't respond to pings, set up the WatchDog timer to
ping it every 24*60*60 seconds, with a fail count of 7. (or suitable
numbers that the GUI will accept).  

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Wed, 15 Nov 2017 07:22:29 -0000 (UTC), Blake Snyder

Quoted text here. Click to load it


It's spelled kludge.  In the not so distant past, I helped maintain a
series of mountain top weather stations.  Service calls were expensive
and best avoided.  As an added bonus, this was in an environment full
of RF pollution.

Quoted text here. Click to load it

Sure, if the watchdog timer is independent of what it's monitoring.
Long ago Kantronics KPC-2 TNC (packet radio modem) had a built in
watchdog timer.  Too bad it was all software and located in the same
chip it was monitoring.  When the KPC-2 hung, the watchdog timer also
hung.  In later models, they simply removed the watchdog timer.

Roll forward a few years, and I'm maintaining servers in a big server
farm.  Remote reboot via ethernet was problematic.  It was quite
common to arrive at the ISP and find a message declaring that the OS
refuses to reboot until some obscure process agrees to die gracefully.
The customer got tired of paying me to reboot his servers, and I got
tried of driving 50 miles to flip a switch.  So, I install a paging
receiver and decoder to initiate a reboot.  That was quite a challenge
as server farms are full of RF interference.  

However, even that didn't quite work.  It seems that most servers have
a "feature" called WOL (Wake on LAN) that allows me to remotely power
on the server.  In order to do that, it needs to have the power left
on to the LAN card(s) even when the server is turned off.  (Note:  WOL
is mostly used for desktops, but at the time was also appearing in
servers).  Sometimes, the ethernet card would hang.  If I reboot the
machine, the LAN card would remain hung.  If I flipped the power
on/off switch on the server, it still would remain hung.  Of course,
with no connectivity, I couldn't do a remote reboot in software.
Compaq later introduced a server management card that provided a
secondary management channel, but it was too expensive.  The only good
solution was to pull the plug on the server.

For server farms, I eventually went to SNMP managed remote power
switches.  I still have a bunch of APC AP9211 switches in service.
<https://www.google.com/search?q=apc+ap9211&tbm=isch
Primary control is via ethernet, but some had a secondary control
channel via the serial port.  

I've tried other schemes and solutions.  Some worked, but all had a
surprise hidden somewhere.

Quoted text here. Click to load it

Won't work and may I humbly suggest that you think about this a bit
more.  The problem is that in any ISO layer cake device, it is
possible to have the lower layers working, while the upper layers are
hung or stuck in a spin look making the lower layers too busy to
respond.  I've seen machines that respond quite nicely to ICMP pings
where the main function (email or web server) is totally hung.  For
these things, you need to test the higher level services and not rely
on the lower levels.

In this situation, there's one big advantage that MIGHT make such a
simple ping work with a wireless link.  All wireless connectivity is
done at layer 2 (MAC layer).  The IP layer is only involved in
managing the device.  If one pings by IP address, it's fairly good
assumption that the underlying layers are working.  Some services such
as SNMP, SMTP for email fault notification, and the usual internal web
server might be hung, but with layer 2 still working, the wireless
bridge would likely still be doing its job.

Now, back to the original problem.  Heartbeat timers and timed reboots
are a kludge.  They're needed because the manufacturer of the wireless
bridge radios didn't do a decent job of keeping his hardware up and
running 24x7.  The failure might be in software, susceptibility to
power glitches, susceptibility to DoS attacks, crappy components, or
environmental issues (overheating).  If the wireless is bridge is so
unreliable that it has to be rebooted once per week, I suggest you
look into the cause of this unreliability, and not apply a band-aid.


--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
Quoted text here. Click to load it


I've got a server on another continent that I need to do an
operating-system reinstall on.  it's a 3 year old supermicro with the
IPMI having a java-webstart based remote console. at some level it
uses RDP, but seems to only work via the java web app.

the IPMI allows viewing the console, turing power on and off, hard reset,
turnging the identity LED on and off, mounting network files as boot
media. etc.  

I'm going to try a PXE boot of Debian install medis from a server in the same
rack.  (the virtual medis boot is poorly documented)

All this remote admin stuff is pretty insecure and, so runs on LAN
addresses, not puclic IP addresses.

The moral of this story, spend extra for server grade SSDs, don't use
laptop drives.

--  
This email has not been checked by half-arsed antivirus software  

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Wed, 15 Nov 2017 07:22:29 -0000 (UTC), Blake Snyder wrote

Quoted text here. Click to load it

You could even have it ping a machine on your network somewhere that you
could take off-line when you wanted to reboot everything.

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Wed, 15 Nov 2017 16:25:57 +0900, ATANARJUAT

Quoted text here. Click to load it

Or query the SNMP "time since restart" and force a rest when the
monitored device has been restarted?

On a cisco router (and lots of other infrastructure tin) you can set
"reboot after xxx minutes"
 - very useful when reconfiguring remotely something that may affect
the in band link the control traffic arrives on
 - as long as you remember to cancel it when you are done......

however a timer with a real power cycle is going to catch "stuff" that
a reset may not - some hardware lock ups have been known to need a
physical power cycle to recover.....

--  
Stephen

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On 2017-11-15 08:25, ATANARJUAT wrote:
Quoted text here. Click to load it

Do you know of a reasonably cheap hardware device that can monitor
something on the network, and powercycle it when needed? A watchdog that
acts on a hung device, say.

I know one or two, but they are expensive.
A timer reboot is too aggressive when a reboot is not needed.


--  
Cheers, Carlos.

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
Quoted text here. Click to load it

I used transitor driven relay wired to the RTS line of a serial port.  
open the serial device and the relay switches on, I used the normally-
closed contacts to interrupt the DC supply to the flakey device (said
DC supply also powered the relay)

A cron job would check the status of the internet and do "sleep 10 <
/dev/ttyS0" when a hard reboot was needed.

--  
This email has not been checked by half-arsed antivirus software  

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On 2017-11-17 10:51, Jasen Betts wrote:
Quoted text here. Click to load it

True, but I don't have any computer on that room.

Have a look at these:

http://www.hw-group.com/products/ip_watchdog/index_lite_es.html
https://www.hw-group.com/products/ip_watchdog/ip-watchdog2_Lite_en.html

They are the exact thing, but expensive for my needs.

There are domotic switches, but those I find are controlled remotely via
a phone app - however, as the device that hangs in my house is precisely
the router, there is no Internet and those switches would not work
remotely. I need something that acts on its own.


I could use an arduino, but I'd have to learn how - which might be
interesting, anyhow.

The best and cheaper seems to be switches than have the chip "ESP8266
sonoff" and can be reflashed. But again, I need to learn the procedure
and the programming.

--  
Cheers, Carlos.

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
Carlos E.R. wrote:

Quoted text here. Click to load it

Use a Raspberry


Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?

Quoted text here. Click to load it

Yes, that's one of the methods I'm considering for when I have the time.

--  
Cheers, Carlos.

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
He who is Peter K+APY-hlmann said on Wed, 15 Nov 2017 14:26:20 +-0100:

Quoted text here. Click to load it

The idea of using a Raspberry Pi would work!

It's actually probably perfect in that it's highly programmable, and it has
all the necessary WiFi, Ethernet, and USB connections!

I did try out the WatchDog feature of all Ubiquiti radios.
Here's my Rocket M5, for example, which has 28 dBm of transmit power into a
30 dBm dish which means I can pickup or throw my Internet for ten miles.
<http://wetakepic.com/images/2017/11/15/Clipboard01c7b07.jpg

The Ping Watchdog facility was unused when I looked just now:
<http://wetakepic.com/images/2017/11/15/Clipboard02.jpg

I turned the Ping Watchdog on, and set it to 86,400 seconds (1 day), and
then set the ping failure number to 7 days.
<http://wetakepic.com/images/2017/11/15/Clipboard03.jpg

The question is what IP address to ping that doesn't exist on the net?
It wouldn't take 192.168.1.256
<http://wetakepic.com/images/2017/11/15/Clipboard04.jpg

So I picked an IP address that doesn't exist on my net of 192.168.1.254
<http://wetakepic.com/images/2017/11/15/Clipboard05.jpg

This only works if you have the login to the radio, which most people on
WISP do not have access to though. So the WISP admin has to set it up.

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Wed, 15 Nov 2017 16:35:15 +0000 (UTC), harry newton

Quoted text here. Click to load it

One of these daze, I should read the Ubiquiti instructions.  I didn't
know it had a built in watchdog reboot feature.  Same warning as with
the Kantronics TNC.  If the watchdog feature lives in the same
hardware that it's trying to monitor, then chances are good that if
one hangs, so will the other.

I've also had the reboot timer in DD-WRT screw up on me a few times,
but that was long ago.  

Bah Humbug (T'is almost the season).

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
He who is Jeff Liebermann said on Thu, 16 Nov 2017 16:03:27 -0800:

Quoted text here. Click to load it

For what I need, the watchdog feature "should" work fine, since if it's
actually hung, I can always manually pull the POE for a minute (or put it
on a weekly or daily timer).

PS: The rains are coming.  

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Fri, 17 Nov 2017 09:58:09 +0000 (UTC), harry newton

Quoted text here. Click to load it

Yeah right.  A few days ago, it was suppose to rain Sunday and Monday.
Yesterday, the prediction was Monday only.  Today, it's no rain.
See the "NOAA Weather Prediction Center"
<http://www.wpc.ncep.noaa.gov
Mouse over the "Day 1, Day 2, Day 3" just above the weather map.
Notice how the southern boundary of the predicted rainfall stops just
south of San Francisco Bay.

Must be a conspiracy.  I want my rain.

--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?

Quoted text here. Click to load it


southern half of the lower 48.  


Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
On Sat, 18 Nov 2017 22:56:12 -0700, nmassello@yahoo.com (Neill
Massello) wrote:

Quoted text here. Click to load it

Quoted text here. Click to load it


Early 2018, NOAA Says"
<https://weather.com/news/climate/news/2017-11-09-la-nina-noaa-updates
<http://www.cpc.ncep.noaa.gov/products/analysis_monitoring/enso_advisory/ensodisc.shtml
Of course, the government (NOAA) is always right.
Grumble...  


--  
Jeff Liebermann     jeffl@cruzio.com
150 Felker St #D    http://www.LearnByDestroying.com
Santa Cruz CA 95060 http://802.11junk.com
Skype: JeffLiebermann     AE6KS    831-336-2558

Re: Rebooting radio in Santa Cruz mountains once a week via GitHub script?
wrote:

<snip>
Quoted text here. Click to load it

A Netonix switch. Single best investment I made for my WISP.

I set them to ping a device on the opposite tower (in a PtP link) so that if the
radio (on this side) hangs (but still responds to pings) it won't fool the
watchdog into thinking everything is fine. The monitored device must actually
pass traffic so that a ping packet can travel to the opposite side, be answered,
and then return to the Netonix, thereby resetting the periodic timer.


Site Timeline