Myrinet against ethernet

Bonjour,

...and, if possible, about the compatibity between 10 Gbit/s Ethernet and Myrinet.

Best regards, Michelot

I suspect the best source would be:

Where I think you will find that _myrinet_ is a proprietary networking technology that is not directly compatiable with 10Gbit/s Ethernet, and that Myricom also offer 10 Gigabit Ethernet products.

rick jones

Bonjour Rick,

Thanks for the link, I should have known it.

"...these Myri-10G protocol-offload Network Interface Cards (NICs) deliver excellent performance at minimal cost, and are fully compliant with Ethernet standards".

What I'm understanding is that.

The word offload seems important, and not easy to translate in French. The Myri-10G is using the same PHY protocol than Ethernet (the same physical level). But, the level-2 protocol is not "MAC 802.3" (MAC Ethernet).

"MAC 802.3" and "level-2 Myri-10G" are sharing the same "PHY Ethernet" protocols, that are the 10GBase-CX4 and the 10GBase-R family.

There is the same thing with IBM FICON protocol avec Fibre Channel. The physical level for FICON is same as the physical level of FC, but the frame level is different.

So, the equipments that interpret the level 2 have to be differencied. A Myri-10G switch is not an Ethernet switch. But we can transport a Myri-10G wavelength as a 10GBase-R wavelength.

It is my understanding at this moment. Best regards, Michelot

Michelot wrote: (snip)

Offload in the case of computer hardware usually means it performs some function that is normally (or previously) done in software.

In the past, some NICs would do the TCP checksum calculation such that the TCP/IP software didn't have to do it.

As networks get faster it is harder for software to keep up.

-- glen

Bonjour Glen,

Thanks for that piece of the puzzle.

Best regards, Michelot

Michelot wrote in part:

Especially some of the finicky housekeeping tasks like endian swizzle, filtering, checksums, stripping headers, packet reassembly,

My ideal "smart" network processing unit (NPU) would require very little OS or CPU support (fewer interrupts). Simplifying:

OUtbound: CPU sends destination IP:socket and pointer to data. NPU opens connection, sends data, checks ACKs [&resends], informs CPU by interrupt (if req) or status register.

Inbound: NPU checks filters (/etc/hosts.allow preload), strips headers, reassembles and drops directly into application memory. Aligned zero copy. Interrupts or flags CPU.

The key is the NPU is a CPU in it's own right, but more highly optimized for stream processing and without high interrupt and protect/mmap overhead. You would still be limited by PCI[e] or whatever bus bandwidth and latency.

Bonjour Robert,

Always in the subject of Myrinet, we can see in this text, page 10, there is a possibility of "kernel bypass". TCP/IP aplications are almost directly on the firmware called MX, without passing through the TCP/IP transport layer and IP.

Euh... where is indermeet? He/she launches this talking and vanishes, perhaps hidden behind a column.

My new and definitive understanding of the interresting Myri-10G network (of Myrinet) is that :

- Possibility of running the proprietary Myrinet Data Layer on a standard 10G Ethernet PHY layer. This set doesn't give a standard Ethernet.

- Possibility of running the proprietary MX layer on a standard Ethernet MAC and 10G Ethernet PHY layer, with the Ethertype 0x86DF (I asked directly to Myricom the value, Ruth is very nice). I learned that Myrinet was a commercial form of the ATOMIC project. So, this possibility is completely conform to the standard Ethernet.

- Possibility to use the proprietary Myrinet network, different at the Data Layer and the Physical Layer from those of Ethernet.

Best regards, Michelot

Thanks for such a wonderful discussion.

I agree the more you bypass the kernel layer (for TCP/IP application) , the better latency you obtain.

Does this mean that the Myrinet frame could be encpasulated in the Ethernet payload . If yes , what advantages does it provide ? anyhow , you would have been using the myrinet NICs which have firmware embedded in them.

Another question, How much is the price difference between Myrinet NIC's and the standard ethernet NICs ?

Thanks

FICON is a protocol that runs on top of Fibre Channel. The underlying frame formats are the same.

Dave

Bonjour dbs,

Thanks for your remark. We can perhaps detail a little bit.

Section 3.1: According to that text, FICON is in the FC standard architecture and uses the FC-SB-2 protocol. This protocol is located in the FC-4 layer. It's a client of the FC-2 (FC-3 seems an empty layer).

In the glossary for example (page 255), it seems that we have 2 cases:

- either the term FICON is refering only to FC-SB-2

- either the term FICON is refering to FC-SB-2 + (FC-3 to FC-0)

Thanks for your advice. best regards, Michelot