[sorta OT] LAN reliability ?

And with USB flashdrive sticks, maybe that's also the way to go for laptops. No user data. Appliance computing.

I think that's where we blow it. 100+k machines around the world. Most data nearby, but all visible.

-- Robert

In article , Al Dykes wrote: :For Windows on desktops on business it isn't so much the MTBF as the :MTTR that makes a happy shop. With all the user data and profile on :the server we just drop in a fresh pre-imaged box when a user has a :probelm, hardware or software.

:IMO MS servers running mainstrea windows applications can be very :good if your expectations o0f scale and complexity are reasonable.

At the risk of becoming increasingly off-topic for "lans", but still on the topic of "user-perceived reliability", I wonder if someone might have some information on reliability of this sort of configuration as a fileserver to replace Novell 5:

fileserver ~~ NFS3 ~~ Linux node ~~~ samba/SMB ~~~ Windows PC

When I was young and impressionable, I saw first hand that NFS (probably v1 then) was incompatible with reliability, because *in practice* a single crashed node that had mounted a filesystem read-write could end up requiring that -all- the nodes be rebooted to clear the problem. And having seen that happen all too often, When I Had My Way in designing our services, NFS was banished. NFS didn't have the best of reputations when lots of different systems were accessing it [e.g, as would be the case for a fileserver.] But NFS has, I know, grown up somewhat since then: has it made it to production grade for reliability, locking, and speed?

Similarly, when I was examining earlier samba editions, locking was a big issue for it as well... as was ability to handle a lot of simultaneous activity. Perhaps that has been cleared up as well?

We don't do much database work, so we don't often work at the level of record-level locks, but we would plausibly have multiple appenders to single files, and we would certainly have multiple people attempting to open the same document with write permissions. With Novell Netware 4/5, the multiple writers issues was handled acceptably for our purposes: a message saying the file was locked was better than two people overwriting each other's changes.

What's the word these days: "Don't worry, those are problems of the past", or "Not there -yet- for NFS + samba?", or "Run Netware Services for Linux to mediate the accesses rather than trusting samba" ?

:Warning: I have been called a "Network Management Bigot" for :requesting all sorts of monitoring. However, my experience has been :that if you look closely enough at how the network is ACTUALLY :running, you will often spot problems before they are manifested :as service outages. Examples range from marginal links which are :reporting only brief intermittent hiccups on their way to total :failure, to routing tables which indicate that the routes in use :are not the routes you designed with the high probability that :when something fails, the network will roll over and die rather :than select an alternate route.

That sounds like much more sophisticated monitoring tools than MRTG or even Fluke's OptiView (nee' Network Inspector). What is available that can give useful information at that level, over a range of manufacturers (e.g., Nortel BayStack, Nortel Accelar, Cisco 3750, likely some Cisco 2960's)?

We presently have only about 25 local switch units (mostly in stacks of 2-4); we might be getting more for redundancy.

More importantly than monitoring software: where can one learn what events can be safely ignored; and what needs replacing "sooner or later"; and what isn't failing now but it's time to tell management firmly that a replacement is a priority? Would there be some good books/documents about network risk assessment?

We are not in a situation where "The network MUST stay up" -- but the longer access is down, the less work our people can do, so it would be better if I could learn how observed problems translate into risk probabilities, so we can do cost/benefit analysis that take into account the then-current financial priorities.

Not necessarily!!! For example, MRTG is extremely useful, as long as you use it to monitor more than just usage. It does an excellent job of monitoring other parameters that are exposed by SNMP, such as link errors, CPU utilization, route changes, and the like.

It also means logging into each router (particularly after a change) and looking at the routing tables, routing topology database, etc., and verifying that everything matches what your design says it should be, then triggering various faults and verifying that the recovery is the recovery you designed. Simple example, routing with EIGRP, is the alternate route a feasible successor? Bottom line, don't assume your design and implementation are correct, prove they are. Then test them to failure so you can be sure they behave correctly under stress.

Careful and routine perusal of the logs can be enlightening. The logs are not for post mortem analysis only. A multivendor environment makes automating the analysis more challenging, but most vendors of "business grade" hardware provide logging of a wide range of interesting events.

Substitute spanning tree for my comments on routing :-) Is the spanning tree what you think it is? Is the topology simple enough that the new spanning tree can be quickly determined when an interswitch trunk fails? Do the "fast convergence" features configured actually work?

Doubt that there is a book available, there is no market for one (I tried, but the mass market is for "pass the cert" books). Part of the problem is that the answer is the classic "it depends." If you really understand how your network works, you will know what is important and what is not. Some examples:

Review yesterday's logs: Can you explain every event reported? Every report of a routing peer associating or disassociating? Every backup call placed? Every spanning tree change?

Still in yesterday's logs: Was the reaction to every network perturbation correct? If a link failed, did it have the proper side effects such as routing peer change reports, dial backup, HSRP active moves, etc.

Still in yesterday's logs: Do the events reported in the log match up with the events reported by your status monitoring system? Are there failures reported by your monitoring system (even "WhatsUp Gold" can do that job) that are not reflected in the log or vice versa?

Look for correlations between the events reported and the traffic, error rates, and the like recorded by MRTG (or whatever). Note that MRTG only gives you 5 minute resolution for the past 24 hours, you may want to look at preserving that data longer if you don't check at the same time every day, including weekends :-)

While you're looking at the error rates, are there correspondences between links which are supposed to be independent? This could imply a common point of failure which needs to be eliminated. On WAN links, it could mean that links which you are paying to have diversely routed have been groomed back to one or more common trunks.

Etc. etc. etc.

The biggest challenge is justifying the effort required to keep track of what is happening. Note that there is significant expert analysis required with little chance of an immediate payback. Be wary of getting caught in the network management success trap. If you do too good a job, users (and their management) will never see all the problems you cured while still beneath their radar, and you'll be asked "Why spend all that money on network management when the network never fails?"

Good luck and have fun!

:Be wary of getting :caught in the network management success trap. If you do too good a job, :users (and their management) will never see all the problems you cured :while still beneath their radar, and you'll be asked "Why spend all that :money on network management when the network never fails?"

As Pooh would say, "Oh, Bother!".

I've been in that trap for years. People see the failures and not the successes or the efforts; and they wonder what you -do-, since you don't seem to be -producing- anything... and then they cut budgets. Then when a disaster happens because you couldn't get the time or money allocated to go redundant or build/buy the proper monitoring and testing tools, it is due to "your Bad Planning". :(

:>More importantly than monitoring software: where can one learn what :>events can be safely ignored;

:If you really :understand how your network works, you will know what is important and :what is not.

I think it would be difficult for me to justify tracking down every CRC error -- test all the cables each time, swap to a different copy of the same kind of NIC, swap to different NICs, SmartBits several adjacent runs in case it is cross-talk related, and so on.

To give an analogy: when our desktop workstations panic once every few years with a memory error, we didn't worry, because we knew that those events could be caused by cosmic rays (literally!) or by natural radiation (the flower-garden borders are granite, and granite is slightly radioactive.) We developed a feel for how often was "normal", and how often suggested a more serious problem (loose SIMM, need to clean the contacts.) There wasn't any point in calling for maintenance on a stray memory error -- the incidence rate, impact, and reproducability were too low.

With a network, I have not yet developed a good feel for "acceptable" error rates. Retransmissions will happen, so single errors or bursts are taken in stride in low levels; it takes a fair number to interfere noticably with the regular data (especially if the link is not particularily under load.) A -stable- error rate is not necessarily a problem.

An error rate is, though, also a hint that the link might deteriorate, perhaps rapidly -- and I don't know how to go from a particular error rate to a likelyhood of real failure. 'This link only has "two months to live"' is going to be treated somewhat differently than 'There is about a 3% chance of this link failing in the next two years, leading to about a half-hour of slow DNS response when it does.' I cannot just go on "gut instinct" to convince management to divert funds to buy a new switch, especially not if any failure would most -likely- happen in the next fiscal year instead of this one...

In article , Walter Roberson wrote: :Recently, I had it put to me that LANs (and firewalls) should be 100% :reliable (barring major equipment failure)

:Is that level of reliability the norm in real SMBs, with 500-ish :hosts, multiple subnets, and a mandatory deny-by-default firewall :policy?

I received some good feedback to my earlier questions -- thanks, everyone.

Today I came across an article that directly addresses real-world IT reliability measures:

- 4.5 hours for network hardware

- 4.7 hours for network transport

- 2.3 hours for security

More figures and number-of-incidents analysis are available from the above article.

-expectations- here are for close to four-nines reliability... on a one-nines budget.]