Hello all, We have a 7507 w/ RSP4/256M and a VIP-2-50 w/ CT3 card running rsp-pv-mz.123-11.T3.bin. The CT3 card is plugged into an Adtran MX2800 DS3 mux. The Adtran is used to consolidate DS1 circuits from various carriers into a single Channelized DS3. So, the layout looks like this:
Carrier A -> DS1 -> MX2800 -> CT3 (7507) Carrier B -> DS1 ---^
For the most part, this has been flawless and very stable up until about a week ago. Our logs show constant T1 controller failures as far back as 3/26/05 and continuing up to the present.
On the Cisco, the DS3 span looks very clean:
CT3 H/W Version: 1.0.1, CT3 ROM Version: 1.1, CT3 F/W Version: 2.4.3 FREEDM version: 1, reset 0 resurrect 0 Applique type is Channelized T3 Description: Muxed DS3 to Adtran MX2800 No alarms detected. Framing is M23, Line Code is B3ZS, Clock Source is Line Rx-error throttling on T1's ENABLED Rx throttle total 306, equipment customer loopback Data in current interval (453 seconds elapsed): 0 Line Code Violations, 0 P-bit Coding Violation 0 C-bit Coding Violation, 0 P-bit Err Secs 0 P-bit Severely Err Secs, 0 Severely Err Framing Secs 0 Unavailable Secs, 0 Line Errored Secs 0 C-bit Errored Secs, 0 C-bit Severely Errored Secs Total Data (last 24 hours) 0 Line Code Violations, 0 P-bit Coding Violation, 0 C-bit Coding Violation, 0 P-bit Err Secs, 0 P-bit Severely Err Secs, 0 Severely Err Framing Secs, 0 Unavailable Secs, 0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
On the MX2800, however we are showing the following errors at the DS3 level in the last 24 hours:
Unavailable Seconds (UAS) 279 Severely Errored Framing Sec (SEFS) 0 Line Coding Violations (LCV) 9 Line Errored Seconds (LES) 334 Line Severely Errored Seconds (LSES) 333 P-Bit Errored Seconds (PES) 53 P-Bit Severely Errored Sec (PSES) 49 P-Bit Coding Violations (PCV) 62 C-Bit Coding Violations (CCV) 0 C-Bit Errored Seconds (CES) 0 C-Bit Severely Errored Sec (CSES) 0 F-Bit Errors (FBE) 77576 Far End Block Error (FEBE) 0
As a result, the DS1 spans on the Cisco look like complete crap, and we have the following in our logs:
Apr 4 20:12:59: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN Apr 4 20:13:01: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP Apr 4 20:13:04: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 11, changed state to DOWN Apr 4 20:13:21: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 11, changed state to UP Apr 4 20:23:15: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN Apr 4 20:23:17: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP Apr 4 20:46:40: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN Apr 4 20:46:42: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 6, changed state to DOWN Apr 4 20:46:42: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP Apr 4 20:47:00: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 6, changed state to UP
Just an example of the Cisco's T1 stats:
T1 2 is up timeslots: 1-24 FDL per AT&T 54016 spec. No alarms detected. Framing is ESF, Clock Source is Line, Rx throttles 1 Data in current interval (79 seconds elapsed): 0 Line Code Violations, 0 Path Code Violations 0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins 0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs 0 Unavail Secs, 0 Stuffed Secs Total Data (last 24 hours) 0 Line Code Violations,2284 Path Code Violations, 0 Slip Secs, 3 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins, 5 Errored Secs, 2 Bursty Err Secs, 3 Severely Err Secs 260 Unavail Secs, 0 Stuffed Secs
Any suggestions on what I should look at? I'm tempted to swap out the cable between the MUX and the Router...