Ethernet over T1 (Ethernet over WAN...)

I would suggest stripping off link headers and encapsulate the network layer in the link layer used for your T1. There is seldom any point in transmitting linklayers since they are used only as "containers on the local media" and has no significance outside the site or LAN.

ppp-over ethernet is a hack that does add an authentification step unavailable in ethernet, besides that it does not add much more then wasted bits

hint: "no ip routing"

Arnold

Look up the function of "bridges" in the LAN environment. Nobody sells them as such any more. Most if not all routers can be configured for "bridging only". An old example of an Ethernet-to-T1 Remote Bridge device would be the Retix 4800 series.

BTW, why are you channelizing your PPP into DS0's ?? Why not look at the T-1 as a single 1.536Mb pipe ??

--reed

Hmmm...Well you do routing on the customer premises to see if the packet is meant for locally connected network or is meant for uplink connection to ISP. If meant for uplink connection, the router *could* send the *same* ethernet packet it received on T1 line (assuming router has CSU capability). Now the ISP router receives an ethernet packet on its T1 line - it either strips the ethernet header and routes the packet on appropriate outgoing interface or (if router supports VPLS or TLS) doesnt strip the ethernet packet and sends it across its L2VPN provider cloud.

Does this make sense....

Thanks for yor response Steve

See thats what my question is - Why cant we use "ethernet" link layer header on T1.

They *do* make a sense. I will give an example - Assume that a company has offices in Chicago, Denver and New York. It runs Ethernet on its LAN. Now, assume that there is a lot of interoffice transaction. Wouldnt it be good to consider all these physically separate LANs as one virtual one (TLS or VPLS). This would have the following benefits:

o At customer premises you dont need to have the *costly* router function to strip ethernet and put another L2 header. You can get away with having just an ethernet switch. o The QoS/VLAN information remains *intact* across ISPs network.

o You basically get an L2 VPN inexpensively.

My point is why do we even need PPP. Why cant we just transmit the ethernet packet on T1 line *as-is*.

Thanks Steve

Precisely, I am looking for a ethernet-to-T1 bridge. In one of my previous replies to this thread, I have mentioned the benefits of doing that.

What if you want to channel your voice calls on some DS0's.

Thanks for your response Steve

you could - but the underlying HDLC processing used by PPP is built into most chips that drive a serial link.

Also - ethernet has some overheads that "waste" bandwidth on a serial link - using PPP replaces 2 * 6 byte MAC headers with 2 to 4 bytes of PPP encapsulation - after not much need for source and destination world wide unique addresses when all you have is the 2 end points on the serial link....

PPP negotiation means that routers have a much clearer idea of the operational state of a link than you would get with the "fire and forget" methods of Ethernet - which assume the reliability of a LAN.

i used to build big bridged WANs using Vitalink / DEC gear - they were always a pain to build and use. As soon as routing became mainstream the vast majority of bridged WANs moved to routing to improve their networks.

The flip side to your WAN scenario is that the serial links have to carry all the layer 2 maintenance traffic which normally occurs within a single subnet - ARP traffic, NetBIOS name broadcasts, any multicasts from spanning tree and so on. On a 100+ user site this can amount to 10s of packets / sec, eating into your T1 capacity.

Routing insulates the link from some of that overhead.

there are serious scaling issues to pure layer 2 environments - the largest i have heard of that actually worked was around 20k MAC addresses, but LANs with 1000s of devices are difficult to maintain because a single problem can wipe out so many users. routing is used within large sites to provide some isolation between subnets

if your WAN uses the "invisible" serial link ineffectively - i.e. all your users in 1 site use an internet feed remotely, or find the wrong active directory server to pull policies from - then your T1 link may be saturated with traffic that could be handled locally.

Routing makes some of the WAN network topology visible to user devices - so that WAN usage can be better optimised (i am not saying that it will be - i spend some of my day job hunting such issues).

Routing is not all that "costly" in the sense you use - after all SOHO NAT routers cost $100 or so, and handle a broadband connection. "enterprise" gear cost a lot more - but there users pay for better reliability, flexibility and support (whether they get it in all cases is an ongoing argument)

Look at the cost of your hypothetical serial links (in the $1000s / month) and the cost of a typical enterprise T1 capable router (say a cisco 2610 - $3k or so with serial interfaces etc). Anything that gets 10 to 20% more out of that T1 capacity pays for itself quickly.

FWIW i agree with some of your arguments - if Ethernet packets can cross the WAN link then you may choose to only route at 1 end, or not route at all (this is a common design tradeoff where Ethernet is used as a WAN or MAN connection, although cutting down on routing makes error isolation more difficult and complicates the LAN design) - but there are tradeoffs, and routed WANs are easier to build, maintain and manage.

QoS argument is a reasonable idea - but most QoS these days seems to be implemented in IP as Diffserv (or done as a local decision within a router queue), and Diffserv can cross a routed network end to end.

Layer 2 QoS might be preserved across an ISP network, but the whole point of QoS is that each device across a path responds to QoS - and current ISPs mainly ignore IP QoS because it doesnt make sense for them to allow some user traffic to queue jump in front of others unless they can charge for that priviledge.

if you want to look at some telco technologies that give layer 2 WANs, try hunting info on Ethernet over MPLS - a good intro will explain some issues that also impact on your Ethernet over a WAN suggestion.

Global VLANs are not necessarily a good thing - there are "only" 4000 unique VLAN numbers in 802.1Q, and some kit puts more severe restrictions on numbers used, so it can be easy to run out of VLANs if they have to be globally unique.

and anyway - i use VLANs to limit where layer 2 traffic can go, not let it spread to anywhere on a network.....

but i dont want that - spanning tree across a WAN means i cant load balance on resilient links.

do you really want to send NetBIOS packets with their fixed sub second timeouts across a WAN where latency means you might end up sending 20 to 30% of the packets twice or more?

PPP is more optimised to a serial link constraints, where the cost of bandwidth is different to a LAN environment.

PPP handles point to point systems more effectively.

PPP understands negotiating useful WAN functions such as link authentication, compression and fragmentation which arent normally an issue with Ethernet.

No.

Welcome

I read the header. And i don't think you understand the issue.

Moot arguments, in fact no arguments at all.

As anyone that has been administrating WAN-links can tell you, route where you canm bridge when you must.

My point, Why not transfer the ip-packets and ONLY the ip packets without extra overhead ??

As extra bonus you will get a reliable network where broadcasts storms ( you are running windows ??) won't propagate across a continent and a predictable network where some degree of control over datastreams are possible.

Network effectivness won't come for free however, it will cost both networking gear and a designer with a clue.

do a google for "RAD TINY BRIDGE"

(works only if the T1 is not broken up into DS0's and only if it is pt to pt - can't use with frame relay, ect)

I have one that I needed for an odd config - cute little box. Has a v.35 on 1 side (plug into your T1 csu) and an rj45 on other side (plug into ethernet). Simple to set up - basically plug and go.

But not a good idea, IMO. I firmly believe in "route where you can, bridge where you must" - makes the network MUCH more manageble.

You're thinking about this backwards. What you're missing is that Ethernet is not designed for global coverage. It's a LAN scheme. As such, it is inadequate for use as a WAN frame. Simple example: the addresses used in Ethernet are merely numbers. They give no clue as to the location of the hosts. Therefore, they are inefficient for sending frames around the world (or even between offices that are geographically separate). You would need an exhaustive list of every MAC address in the world to route packets globally using nothing but Ethernet frames. This list would have to be installed in every router.

In order to move packets around the world efficiently, you need a network layer protocol such as IP, and routing protocols to go with it.

So the way to make this work is to encapsulate the *meaningful* information, i.e. the IP packets, directly inside the frame structure used by your WAN link. That's what PPP does. And more, it also can be used to dynamically assign IP addresses to the local hosts. And you toss out the irrelevant overhead, which is the Ethernet overhead that happened to be used in the originating local network.

To put this another way, if we did as you suggest, toss out PPP and just send the Ethernet frame intact over a WAN, then how would the WAN know where to route that frame?

Answer: it would only know if the WAN were set up as statically configured tunnels, or dedicated links, between point A and point B. Well, what's so clever about that? Can you imagine how complicated the administration of the global infrastructure would be if every WAN link were a statically configured, dedicated link?

And can you imagine how inefficient it would be to see ARP broadcasts going to all hosts attached to this huge Ethernet?

I remember a few years ago, when some folk thought it imperative to send FDDI frames intact over SONET. As if the FDDI frames had some intrinsic value that had to be protected at all cost. Never mind the issue of sending tokens (potentially) all over the world, creating ridiculously big ring nets. This is the same sort of thing. It makes no sense to make a big deal about the Ethernet frame, outside the confines of a *local* area network. What counts is the IP packet.

If you want to build a protocol of your own directly over Ethernet, that's fine too. But now you're going to be limited in how far the frames can travel, unless you go to the trouble of reinventing the wheel and creating another network layer protocol. Still, in specific circumstances, it might make sense to create such a home-grown protocol, as long as you're aware of its limitations.

Bert

You're thinking about this backwards. What you're missing is that Ethernet is not designed for global coverage. It's a LAN scheme. As such, it is inadequate for use as a WAN frame. Simple example: the addresses used in Ethernet are merely numbers. They give no clue as to the location of the hosts. Therefore, they are inefficient for sending frames around the world (or even between offices that are geographically separate). You would need an exhaustive list of every MAC address in the world to route packets globally using nothing but Ethernet frames. This list would have to be installed in every router.

In order to move packets around the world efficiently, you need a network layer protocol such as IP, and routing protocols to go with it.

So the way to make this work is to encapsulate the *meaningful* information, i.e. the IP packets, directly inside the frame structure used by your WAN link. That's what PPP does. And more, it also can be used to dynamically assign IP addresses to the local hosts. And you toss out the irrelevant overhead, which is the Ethernet overhead that happened to be used in the originating local network.

To put this another way, if we did as you suggest, toss out PPP and just send the Ethernet frame intact over a WAN, then how would the WAN know where to route that frame?

Answer: it would only know if the WAN were set up as statically configured tunnels, or dedicated links, between point A and point B. Well, what's so clever about that? Can you imagine how complicated the administration of the global infrastructure would be if every WAN link were a statically configured, dedicated link?

And can you imagine how inefficient it would be to see ARP broadcasts going to all hosts attached to this huge Ethernet?

I remember a few years ago, when some folk thought it imperative to send FDDI frames intact over SONET. As if the FDDI frames had some intrinsic value that had to be protected at all cost. Never mind the issue of sending tokens (potentially) all over the world, creating ridiculously big ring nets. This is the same sort of thing. It makes no sense to make a big deal about the Ethernet frame, outside the confines of a *local* area network. What counts is the IP packet.

If you want to build a protocol of your own directly over Ethernet, that's fine too. But now you're going to be limited in how far the frames can travel, unless you go to the trouble of reinventing the wheel and creating another network layer protocol. Still, in specific circumstances, it might make sense to create such a home-grown protocol, as long as you're aware of its limitations.

Bert

First let me thank you for such a detailed and informative reply!

Hmmm...

Understood.

I havent worked with real-life networks as such so I really didnt know the paradigm - "route when you can, bridge when you must". Now, when I think more about it, it does make sense to use routing in WAN.

Makes sense.

Here, you must be referring to the headaches in allowing LAN to expand beyond its physical boundaries.

Frankly speaking, I didnt really buy the costly argument myself when I read it in "Ethernet over SONET" white paper from PMC-Sierra dated Feb, 2002

"Often, the inter-working function must terminate the Ethernet and map the underlying IP traffic into a new Layer 2 (L2). Alternatively, the Ethernet is encapsulated within another L2 technology. Both of these techniques introduce additional complexity and cost into the WAN interface".

Slapping a different L2 header may not be as complex after all.

I understand your point...

Well, I agree with you again here. Even with Ethernet L2 QoS (802.11p), preserving it would make sense if the provider transport network cam honour those bits i.e. have its MPLS QoS bits derive from Ethernet QoS.

Sure.

I like this argument.

In a VPLS scenario, we would need to run to spanning tree protocol on L2 switches (as you indicate). Although it has the disadvantage of having long convergence time, and non-utilization of redundant link, it may be a *cheaper* option - in terms of cost (of service/equipment).

Very well.

thanks again for an edifying response, ;-) Steve

Thanks for pointing this out! Steve

By Ethernet on WAN, I mean it would be either used as in VPLS/TLS scenario (for enterprises) or just as an access technology for the end-customer (where ethernet frame from users network travels on WAN link to the PE router of ISP where its ethernet header is stripped and the IP packet is routed).

Understand the argument about overhead and managing the Point-to-Point link parameters (managing link-layer and network-layer besides authentication).

As I mentioned earlier, WAN router would do L2 switching (basically sending L2 frame across its network encapsulated as an MPLS packet) or the router would strip ethernet header and perform routing.

Well, for VPLS case, the WAN router would perform MAC learning etc. (so no static config) and there is no need to have a dedicated channel in provider cloud for this traffic.

Agreed, but over a period of time ARP flood is going to decrease.

Well, tell it to VPLS/TLS folks ;-) I kinda dont buy VPLS/TLS argument too much. Not sure how many active deployments are out there for it.

I didnt have any such intention ;-)

Anyways, thanks for your response! Steve

It's a bit late in the thread and much has already been said. Parts were really interesting, thanks guys. There is, however, a bit I think I can say something about.

On 2004-10-18, Mark wrote: [snip]

You forget to list the negatives, such as making those lans one big fat broadcast domain. No, switches won't help there. Yes, your offices all run machines that like to broadcast. Lots. And all that broadcast traffic _has_ to go over the T1. It might mean a broadcast storm for every single desktop that crashes or gets switched on, for both sides of the network. What is the cost of that in wasted bandwidth? Nevermind that DHCP also will be bridged over the T1, latencies increase, whatnot. You Really Do Not Want To Go There.

The router function is not so costly if you look at all the other costs. You still need people to manage the network. Somethimes things just break. Then you will want to find out what went wrong or at least what to do to make it work again, and do so *quickly*. You cannot do everything with a switch. If your operations people are even just 10% quicker in fixing things with a router than with a switch, the router may be well worth the extra cost. I'll leave the actual math to you, as it is complex and has too many factors off-topic to this group.

Someone else already answered this. You could, but you don't really want to: it doesn't make much sense. Money isn't everything there is to managing networks. To give just one other: Making it work, and making it work well. How reliable do you want your network to be? Then you might as well use technology that has explicitly been designed to do the task, and not something else that was designed for something else entirely. To pull another counterexample back into this: Even PPPoE doesn't *replace* ethernet, it takes ethernet and adds something to it that wasn't there before. *Replacing* ppp with ethernet frames loses things that are useful on a point-to-point link and adds things to it that don't make sense on the medium. Since T1s are rather limited, even compared to 10Mbit ethernet, those few extra bytes are now dead weight that may prove to be expensive.

What exactly you lose and what exactly you (don't) gain is an interesting question that is unfortunately better suited to a networking course. You could try and read _Computer Networks_ by Tanenbaum for an overview of the issues involved.

While it's certainly possible, using GRE, to carry ethernet frames over any IP system, there's rarely a need to carry the extra overhead of the ethernet frame, which is only required on the local lan. PPP by itself, is capable of carrying various networking protocals (not just IP), so there's really no need for carrying ethernet frames.

Given that the purpose is to move data, not just ethernet frames, no it doesn't. There are many methods available, to move network data over a telecom link, that are better suited to the task, than ethernet.

Why bother? What's so magical about ehternet headers?

The entire world is heading to IP, including the phone companies. What does ethernet over T1 get you that can't be obtained by the various IP methods.