WAN Ethernet

Thanks Rich for that clear reply

Best regards, Michelot

Bonsoir Steinar,

Redundancy without protection, what does you mean? Bridging the signal over multiple paths (it is redundancy) need necessarily switching at the other end, and it is SNP protection.

(1) The provider has surely protection (2) SDH is a transport network protocol, and WDM is a modulation. You can transport SDH signals on a single wavelength or multiple wavelengths.

If all is ok, if there is no failure path for example.

Latency is not made only of optical transmission delays. Travel through equipments is also important.

You have an example in SAN, with e.g. Fibre Channel that needs around

20 =B5s between a transmitted client frame and the acknowledgement from the remote side. To limit the treatment latency a special GFP mode is used in the source and sink SDH equipments.

At the source, a configurable quantity of FC characters could be encapsulated in the GFP frame, almost at the physical level, without waiting the full FC frame is received prior to be injected in the GFP payload.

And intermediate nodes add very low latency.

Best regards, Michelot

Bonsoir Stephen,

Note that today GFP is commonly include in all new SDH equipments, since around 2002.

regards, Michelot

Let us say we have 3 cities, call them A, B and C. We would like to have redundancy. We have found that building the redundancy ourselves using routers and point to point links *without* protection on the underlying circuits (AB, AC and BC) works better for us than having the circuit provider giving us protected circuits. Yes, our circuit providers offer SDH circuits without protection.

As mentioned above, the providers can also offer us circuits without protection. The normal case here is that we buy two types of products, SDH capacity (normally comes with protection) and WDM capacity (which normally comes without protection).

At least in this country the circuit providers differentiate between SDH capacity and WDM capacity as I mentioned above (but WDM capacity is often delivered on an type SDH interface).

I can only speak for the company I work for - we don't see any significant difference between SDH equipment and Ethernet equipment here.

As I said - speed of light in fiber is the dominant part of the delay here. For us the added delay at intermediate SDH nodes versus router hops is completely irrelevant (as long as there is no queueing delay in the routers). We don't do SAN.

Steinar Haug, Nethelp consulting, snipped-for-privacy@nethelp.no

Bonsoir Steinar, :

So, your bandwidth is twice you need. And, if there is a problem, you can't distinguish if it is from you (client layer) or from the provider (server layers).

For each point to point link, what is the availability announced by the provider, how is it controlled by the client, what is planned if the traffic contract is not respected? All important questions to be asked.

Protection can be transparent for the client, the provider offers circuits with QoS.

So, I understand that SDH delivered on WDM is protected at the SDH level and not protected at the optical level, it's not a fully OTN network.

You have to put together 5 =B5s / 1 km for the transmission in the fiber, and around 100 =B5s for the duration of a 1500 bytes-frame at 155 Mbit/s.

With that system, it can be some curious effects. For example, if the link AB fails, the protected circuit AC can fails also. You are not sure that the provider uses the same path for the both links.

Regards, Michelot

Bonsoir Steinar, :

So, your bandwidth is twice you need. And, if there is a problem, you can't distinguish if it is from you (client layer) or from the provider (server layers).

For each point to point link, what is the availability announced by the provider, how is it controlled by the client, what is planned if the traffic contract is not respected? All important questions to be asked.

Protection can be transparent for the client, the provider offers circuits with QoS.

So, I understand that SDH delivered on WDM is protected at the SDH level and not protected at the optical level, it's not a fully OTN network.

You have to put together 5 =B5s / 1 km for the transmission in the fiber, and around 100 =B5s for the duration of a 1500 bytes-frame at 155 Mbit/s.

With that system, it can be some curious effects. For example, if the link AB fails, the protected circuit AC can fails also. You are not sure that the provider uses the same path for the both links.

Regards, Michelot

SONET/SDH APS requires twice the bandwidth as well due to the protect circuit.

Of course you can. If a circuit goes down, you see it on the routers.

The SLAs are typically the same for "unprotected" circuits as for "protected" ones. The "client" has no control over availability in either case, just credits for SLAs not being met. However, in my experience, buying an SLA will not appreciably change the actual availability, but it will often increase the circuit cost more than the maximum credit you can recover, so buying an SLA is usually counterproductive.

Circuits fail, whether protected or not, whether there's SLAs or not.

It may be. My experience is that if one side of a "protected" SONET/SDH circuit is cut, the other tends to be cut as well due to telco incompetence (improper provisioning, grooming, etc.).

Given that a "protected" circuit costs more than twice what two "unprotected" circuits do, it's cheaper to buy unprotected circuits and handle failures yourself, provided you're running packet traffic and not TDM. People using TDM have no real choice.

No, there are four different services: protected SONET/SDH circuits that are most likely transported over a WDM network, unprotected SONET/SDH circuits that are most likely transported over a WDM network, raw WDM capacity that itself offers no protection, and raw dark fiber that itself offers no protection.

If one wanted, you could run your own SONET/SDH gear over the raw WDM capacity or dark fiber and use APS, but why bother? Push pure Ethernet or POS over the WDM links or dark fiber and you get double the bandwidth for less cost (except when there's a failure, when you get the same bandwidth for less cost).

No, it's not, but as consumers we don't care. All I care about is that my bits get from point A to point B for the lowest cost. I'll handle reliability myself, since I can do a better job than the telco can (speaking from experience).

One of the problems is, at least in the US, that the "working" and "protect" paths are often provisioned through the same ROW -- if not the same fiber bundle. When you lose one, you usually lose the other, so APS buys you nothing in most cases.

If both unprotected p-t-p links are cut, an APS-protected circuit would have died too, so there's no difference in failure mode there. If only one p-t-p link is cut, the packet network can deal with that, though it's an order of magnitude slower than APS would handle it. However, that's outweighed by the fact that when there's no cuts, you have twice the bandwidth available at a lower cost than the APS solution.

Now, this does lead to people being sloppy and buying half the bandwidth they really need to handle failures, but that's a human problem, not a technical problem.

S

it depends on how much improved availability is worth.

availability is often much better for protected circuits when they are routed separately thru the carrier cloud.

agreed - but a lot of more modern telco networks using SDH build assumptions into the operations model

the telco may well assume that all "important" circuits are protected - and take down 1 of the 2 paths for upgrades during normal operations - which might be where all your "hits" come from.

incompetence

you need a better telco :). FWIW at work we dont offer unprotected circuits in some circumstances - eg. E1 links - the hassle of having different types of config in operations isnt worth the minimal bandwidth saving / incremental costs.

Not true (or not always true) - it depends on the protection. A well designed SDH network should have ring structured paths wherever feasible.

Most of the overall circuit cost is in the "last mile" once you are having to build a circuit anyway, since incremental costs on SDH are low across the shared bits. so the Q is usually where is the SDH protection implemented - it should usually be the 1st SDH mux at the edge of the carrier cloud.

Protection options (or lack of protection) gets more flexible for faster circuits - there are fewer circuits and each costs more, so customisation makes more sense.

in practise it is more complex than that.

An SDH network will probably cross several different physical layers - and some of those may well be WDM of some description (it gets more attactive as the distances increase, so last mile access fibre is often not WDM).

if you look at it as "OSI model" layering, then WDM is layer 1, and SDH is layer 1.5 or 2.

it depends on what else you need. SDH is still a good way to get fast healing from a break - the spec says 50 mSec, but modern muxes are faster than that. Ethernet / IP equipment can switch just as fast, but tuning it to get that convergence time isnt always simple.

the risk is if you have 2 different circuits, then the telco is likely to put them on the cheapest connection - diversity always costs money. So, 2 circuits from the same site arelikely to follow the same last mile route unless you can specifiy they dont in some way.

It is common in Europe to have highly available connections with diversity / seperacy (never do remember which is which) - this gets you 2 fibre paths to

2 PoPs.

Really paranoid users insist on dual muxes as well.

Agreed - i am working on a network with unprotected GigE over SDH. But the same pipes carry SDH protected links for other services (such as ATM and E1s for voice).

Bonjour Stephen,

Yes, you're right, it's view like that at the client side. Now, the provider can use the protected path to carry a low priority traffic, which can be removed by a upper priority traffic, in case of failure.

...and you don't have traces to know if the problem comes from the transport network, or from the access to the transport network. I understood that it was CE routers, and not PE routers. So you need to join the POPs.

SLAs depend on the statistical duration of the service interruption, and this duration must be better when there is a protection (switching less than 50 ms in case of failure).

It depends on the client. A correct SLA contract include the penalty rates, and the client can ask to apply those each time the SLA is not respected. Today, there are some tools to measure that precisely, at the client side.

It depends on the protection types. For example, regenerator section protections are very common in SDH microwaves.

You have also a direct protection at the WDM level, through the OTN framing. And, when SDH is over OTN, the SDH protections can exist independantly of the OTN protections.

Long distance providers have MNT devices highly capable. This don't avoid the protections at the client level, to still improve the QoS, and protect the access.

Designing a protected transport network is a real job.

It's not really the case in rings.

Best regards, Michelot

Bonsoir stephen,

Thanks for all your good comments.

No, SDH is layer 2 (frame level) and layer 1 (G.705 for electrical interfaces, G.691 for optical interfaces, and F.xxx for wireless interfaces).

WDM is a modulation, not a protocol. Do you say that Manchester is the full PHY level?

In SDH over WDM, you have SDH (level 2 and 1) with the WDM modulation. In SDH over OTN, you have SDH (just frame level) mapped in OTN (level 2 and level 1 with WDM modulation)

In Ethernet over GFP over SDH, you have Ethernet (in fact MAC level, that is level 2) over GFP (level 2 adaptation) over SDH (level 2 and

1).

I agree also

Best regards, Michelot

i think i worded this wrongly. what i meant was that SDH is can be layered over WDM, so although you can buy both services, they are not directly comparable.

when i go and buy a WDM system then from a deployment perspective i am adding a "layer" to my network design - it has a function comparable to a shim layer between physical fibre and SDH.

OSI is meant as a reference model - the layers often dont map directly to implementation, and may not exist, or be "split" into multiple sublayers.

agreed - but in my network in some places i have IP over Ethernet over GFP over SDH over DWDM over fibre - 3 OSI layers for 6 system "layers" for lack of a better term.

the very fact that we have to describe which bits of OSI layers different parts of an implementation map to makes the point....

The thing is, the customer has no control over that -- not even visibility into when it's been done.

The US Federal Reserve recently did a study on diversity and found that it cost nearly $100,000 _per circuit pair_ and over six months to determine if they were routed diversely or not. They only had the budget to check four circuits out of the twenty they cared about, and it happens that all four were in fact diverse -- but the carriers admitted they had not marked them for diverse routing and in fact had made no attempt to do so. It was pure chance -- and it might have changed by the time the results were written due to grooming.

Nearly all customer-visible hits can be traced to human error during provisioning or backhoe fade.

Here in the US, all circuits except SONET are assumed to be unprotected unless you specify protection or the SLAs you pay for make protection worth the carrier's costs.

The thing is, buying multiple unprotected circuits from multiple poorly-designed carriers will be cheaper than buying a single protected circuit from a single well-designed carrier -- and it has the additional business advantage that you can add and remove carriers without much impact (particularly important when half of them go bankrupt every year). Will availability be better or worse? My experience is better, but that's open for debate.

Roughly two thirds of all outages I've experienced were due to failures (usually backhoe fade) in the last mile. The other third is due to failures in chokepoints (e.g tunnels and bridges) where carriers are effectively forced to put the working and protect paths into the same ROW. APS buys you nothing in either case.

Even with unprotected circuits, the carriers do make a reasonable attempt to reroute your circuit around failures if there's spare capacity. In the cases where they can't, most/all of their "protected" circuits are down too.

Customization depends on your total monthly bill, not the cost of individual circuits. I can buy a hundred OC-3 links and get more flexibility from a carrier than a customer ordering a single OC-192. But, in general, I agree with your statement.

PPP, HDLC, or Ethernet is layer 2 in my world, so everything else is below that.

The OSI model is largely irrelevant outside the classroom anyways; it was a marketing exercise by ISO to compete with IBM's 7-layer model for SNA.

It's virtually impossible to get convergence times under a few seconds with IP; however, most applications/users can accept that as long as it doesn't happen very often.

That's virtually unheard of in the US unless the customer pays the cost of trenching themselves. The only "common" case was for Wall St in NYC, and

9/11 managed to take out _both_ COs for that area. Only companies that paid for circuits to a _third_ CO in a different exchange survived -- and I'd bet that those were different circuits, not a third stop on a SONET loop. S

I'm guessing this refers to a cable break due to someone digging a hole. But I don't understand the word "fade".

Difficult to believe that it costs so much...

Perhaps the four circuits were in the same section protected fiber.. and the US Federal Reserve could have save money.

But above in your post you just said "The thing is, the customer has no control over that -- not even visibility into when it's been done".

Interresting

It's correct today e.g. with the virtual concatenation.

You can add, below or at the same level. Final layer 2 can be an aggregation of different sub-layers 2.

Of course, whatever the reference model, some exceptions exist, but we can analyse and justify the gap of the exceptions because we have a reference.

Personaly, I'm always a pupil for all I don't understand.

All transport network can be modellised in layer networks, as it is specified in G.805 (for oriented connection) and G.809 (added information for the connectionless transport networks).

For each layer network you have a specific protection. And, here, you referred to the IP layer network protection and not, e.g., to the ODUk layer network protection. Convergence times are different and depend on the layer network.

Regards, Michelot

Personally I find the ISO OSI model useful as a *conceptual* model, and at the office we often talk about L2 versus L3. But GOSIP and similar attempts to legislate the use of ISO OSI *protocols* failed rather spectacularly - in this respect I agree it is largely irrelevant outside the classroom.

For each layer you *may* have specific protection. If I buy an unprotected STM-16 wavelength between A and B, and run IP over this, and don't add any other circuits for protection - I have *no* protection at the IP level. That would not be very smart, but it's certainly possible. (Please don't tell me that the STM-16 wavelength provider always has protection on the wavelength - we know for a fact that is not the case.)

Steinar Haug, Nethelp consulting, snipped-for-privacy@nethelp.no

I'm not really sure where the term "backhoe fade" came from or how long it's been around, but it's used pretty commonly in the IP network operator community to refer to fiber cuts due to digging/trenching.

My best guess is that it was an abuse of "rain fade", which is slang for problems microwave and laser systems experience with inclement weather.

S

*shrug* That's what they said the project cost.

Keep in mind they were dealing with transcontinental circuits, tracing every single segment through multiple levels of sub-carriers swapping fiber with each other, down to physical trench locations, manholes, etc. The costs were mainly manpower, since up to a dozen carriers were involved for each circuit and none of them had compatible mapping systems or databases. If you don't go to that level, though, there's lots of ways a circuit can cross even if it looks diverse on paper.

Countries with a single monopoly telco/PTT probably don't have it quite as bad, but it's still not an easy problem.

They all went to radically different places.

I meant that even if you order protection, there's no way you can actually make them implement what you ordered, nor can you determine if they did. The carriers themselves are incapable of doing so -- paying out on SLAs is cheaper than doing it right.

I meant when relying on IP-level mechanisms for rerouting, sorry.

S

The thing is, the rates customers pay for protected circuits are based on the premise that both paths are dedicated to that circuit. If the telco is using the protect path for someone else's unprotected circuit, they effectively get to sell the same bandwidth twice.

A customer doesn't care where the error is, exactly; knowing that doesn't get their bits through. Since they're paying for the circuit from CE to CE, when it goes down they just call the telco and yell until it's back up again. What does colocating CE equipment in the CO/POP help with?

True, but irrelevant. I can get the same SLA (say, 99.9999% uptime) on an unprotected circuit as a protected one. The credits might not be the same, and the number of times you invoke it may be different, but the other terms will be identical. The sales people don't care, and management doesn't either; they price their circuits with the assumption they'll have to pay the maximum credit, and if they don't it's just extra profit.

Measurement isn't the issue. The issue is that every telco I've dealt with caps the maximum credit you can receive, so that even if the circuit is down

100% of the time, you still pay 60-80% of the monthly fee (depending on how good you are at negotiating with their salesperson).

Doing it right requires hiring talented people and buying good equipment, both of which are expensive. It's a lot more profitable to provide mediocre service.

You might want to study some high-profile fiber cuts in the US. There have been several cases where tunnel explosions/fires have cut off entire cities from the phone/data network; ditto for train derailments (Sprint put all their fiber next to railroad tracks, since they were owned by Southern Pacific Railroad, and other carriers use that fiber too). You'd think nobody but the telco ops folks would notice, but they make the headlines every few months because both sides of a ring were put in the same ROW.

My favorite is an incident in Houston where one guy trenching with a backhoe managed to cut _every single copper and fiber cable_ going into a CO which also happened to be the tandem for that side of town. About a million people were without phone service for nearly a week. Apparently the CO only had a single entrance, so from the far side of the road where the two sides of the rings met over to the CO wall, they were vulnerable. Backhoes are drawn like magnets to such mistakes, and they're all over the place if you look closely.

S

Stephen Sprunk wrote in part:

Some fear The Backhoe. I rejoice in it's coming.

For after it has cut its' fill of cables, customers learn to appreciate the service they had taken for granted lo these many years. And come to appreciate the butt-cracked cabler, who connects them to the blessed service.

Murphy's Law in-the-steel who reminds us of our mortality, and the folly of five-nines.

-- Robert