FCIP issues with SAN replication

Meanwhile, at the comp.dcom.sys.cisco Job Justification Hearings, Chandler Bing chose the tried and tested strategy of:

My first instinct would be to simulate the 1G WAN by bringing the two units together and linking them with a simple gigabit link [or even 100M to simulate a worst case scenario], and working upwards in complexity from there. Easier said than done, of course. When you get it working you can have a look at a packet capture to give you a rough idea of what it should look like.

Rest assured that the second point will be addressed as soon as you address the first one :-)

Vendor support have been known to be correct on occasion so I'll reserve judgement for now.

How far apart are the two units being replicated? Minimum ping rtt is the best distance metric.

Reply from 208.69.34.231: bytes=3D32 time=3D49ms TTL=3D56 Reply from 208.69.34.231: bytes=3D32 time=3D51ms TTL=3D56 Reply from 208.69.34.231: bytes=3D32 time=3D48ms TTL=3D56 Reply from 208.69.34.231: bytes=3D32 time=3D48ms TTL=3D56

So in this case the min rtt is 48ms.

This may be relevant:-

FWIW FCIP is a standard protocol - if HP have written something non standard then they should have it documented....

Note a sniffer would normally refuse to decode something it doesnt understand, unless whoever wrote the protocol didnt follow whatever escape clauses are built in to allow non standard formats inside the standard wrapper.

If you were being paranoid you would find a HP analyser and see if that shows errors.

a Cisco doc about designing with FCIP

FC seems to use a guaranteed buffer scheme, where there needs to be enough buffering to cope with the path delay to get wire speed throughput.

i have run into issues with buffer credits in FC switches, where you need enough to cope with the speed / delay .

As OPs have commented - check the timing, The GigE link may not follow a direct route, so you may have more delay than you or the protocols expect.

Bingchose the tried and tested strategy of:

HP did something similiar in a lab, they setup the MPX's on a single switch (No WAN) and had them replicating at break neck speeds. When we pointed out that there was no WAN delay, bandwidth limitations, or other devices in the mix they simply shrugged at us.

We have implemented QoS which seems to have given it additional bandwidth (not sure why). We've seen it climb to 45Mbps, but then fail after 12 hours.

The two FC gateways and SANs are 1300~1500 miles apart, RTT ping times are ~36ms with very little jitter.

Alas, the article you linked has no solution. I found several like this, which we pointed out to HP.

I since gotten Wireshark to decode properly. I had to disable FCP decode.

I suspect that's at least part of our problem, as I believe the SAN replication traffic over FCIP is synchronous. The QoS policy we implemented helped, but still fails with high TCP timer expired error count. I have nothing to correlate this to in my wireshark captuers though.

If the FC switches have a buffer configuration, I'm not familiar with it and I'm relying heavily on HP and our SAN engineer to configure those pieces. It is interesting that we have little to no visbility into the FC switch and any errors occuring on that side. The focus seems to be on the ethernet and network side.

Now the update:

I discovered if I disable the FCP decode, Wireshark does decode it correctly as FCIP.

We applied a QoS config to flag SAN replication traffic as DSCP EF and have seen consistent ping times of ~36ms between sites and the bandwidth climb as high as 45Mbps on a 1Gbps link. They still fail after replicating for a few hours. Last time we watched them replicate for 12 hours and then fail. The TCP timer exceed counter seems to indicate that is the problem, but I have nothing significant on the wireshark captures to support this.

HP has decided that the MPX110 on the far side needs to be replaced. I'll post an update after that's done.

Well. As expected, I guess your issues are likely being caused by the round trip delay.

HP have been pretty lame in re-creating your environment since they have failed to emulate the RTT delay.

Perhaps they could consider using nistnet? This is a WAN emulator which runs on linux and very nicely creates a WAN in a Lab.

Obviously you will need a decent computer to run nistnet if you want GBE performance. I last used it about 4 years ago and ran it on an HP DL 360 G3 server using the two built in GBE interfaces. I forget now the bandwidth we needed but it may well have been GBE.

I see that there now seems to be something called WANem mentioned in the googlesphere. Might be worth a look. Nistnet for sure works a treat but has been around for years 'n' years and needs a bit of linux knowledge to get to go.

I have never used FC in anger and don't know much about it however my guess is that there is a VERY good reason that is is not that widely deployed. Everything I have ever heard about it makes the hairs on the back of my neck tingle alarmingly. For example, I would bet that Google don't use it AT ALL in any of their data centres.

Of course loads of people use FC, but loads of people used token ring too despite it being completely crap.

If you like I will consult on this for you and get to the bottom of it. Email in headers does work although I don't read it very often and it fills up with spam. Send me mail and perhaps post here mentioning when the mail was sent in case I miss it among the junk.

OK let me explain. TR was crap because every single conputer in the ring had to correctly process every single bit communicated. A single bit error by ANY single computer in the ring resulted in a lost packet. Twisted pair (or of course yellow "garden hose") Ethernet is a hundred of times more robust. Token ring relied on Chinese whispers in order to work at all. It was simply a daft idea driven by effective sales and marketing. No one ever got fired for buying IBM:)

replying to Chandler Bing, John CG wrote: Hi! Reading all the dramas on this thread is kind of entertaining.

Seriously, this thread is 5 year old. Now we have 2G FC disks for 10 dollars each. I have done hard drive speed test, and these 10-dollar disks sends data easily over 100MBytes per second over a fiber link to my linux box, 800Mbps to saturate your 1G FCIP link.

Why didn't you first take out the FC switches and test the raw 1Gbps link? The test will need 2 of the 10-dollar hard drives and 2 linux computers each with a QLogic adapter. Qlogic adapters are 5 dollars each. You should be able to copy a

If that could not be achieved, the FCIP link is the first problem to solve. If that could be achieved, then move your disks to the home-grade linux boxes is the solution.