trunk connection dies when 6509 reboots.

Post output of

show version

show interface status

show module

show diag

show spanning-tree - for both the 6509 and a troubled switch

Are all troubled ports on one 6509 module ?

How many troubled ports are there in total ?

I've seen a similar problem on a 3750 switch, but it would happen when plugging PoE IP phones into the ports. The phones would come up most of the time, but sometimes if we connected/disconnected the phone a couple of times the port would not come up at all. If we did a shut/ no shut on the port in question it would come back up and run normally. I have not figured out what was causing this problem yet either. Maybe the next time you see the problem you could try the shut/no shut and see if that brings the trunk back up.

3 7

problem switch: =================

sh ver Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version

12.2(25)SEC, RELEASE SOFTWARE (fc2) Copyright (c) 1986-2005 by Cisco Systems, Inc. Compiled Fri 15-Jul-05 02:03 by antonino

ROM: Bootstrap program is C3750 boot loader BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEC, RELEASE SOFTWARE (fc4)

MDF uptime is 16 hours, 15 minutes System returned to ROM by power-on System restarted at 18:24:24 EDT Sat Sep 15 2007 System image file is "flash:c3750-ipbasek9-mz.122-25.SEC/c3750- ipbasek9-mz.122-25.SEC.bin"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption. Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

If you require further assistance please contact us by sending email to snipped-for-privacy@cisco.com.

cisco WS-C3750G-12S (PowerPC405) processor (revision N0) with 118784K/

12280K bytes of memory. Processor board ID CAT0938R5D2 Last reset from power-on 3 Virtual Ethernet interfaces 12 Gigabit Ethernet interfaces The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address : 00:15:C6:0B:5B:00 Motherboard assembly number : 73-9678-05 Power supply part number : 341-0048-03 Motherboard serial number : CAT09390DRH Power supply serial number : DTH09312UUS Model revision number : N0 Motherboard revision number : A0 Model number : WS-C3750G-12S-S System serial number : CAT0938R5D2 Top Assembly Part Number : 800-25856-02 Top Assembly Revision Number : C0 Version ID : V04 CLEI Code Number : CNM8ZV0GRB Hardware Board Revision Number : 0x06

Switch Ports Model SW Version SW Image

------ ----- ----- ----------

----------

• 2 12 WS-C3750G-12S 12.2(25)SEC C3750- IPBASEK9-M

Configuration register is 0xF

MDF# sh int status

Port Name Status Vlan Duplex Speed Type Gi2/0/1 xxxxxx connected trunk a-full a-1000

1000BaseSX SFP Gi2/0/2 xxxxxx connected trunk a-full a-1000 1000BaseSX SFP Gi2/0/3 xxxxxx connected trunk a-full a-1000 1000BaseSX SFP Gi2/0/4 xxxxxx connected trunk a-full a-1000 1000BaseSX SFP Gi2/0/5 xxxxxx connected trunk a-full a-1000 1000BaseSX SFP Gi2/0/6 notconnect 6 auto auto Not Present Gi2/0/7 notconnect 6 auto auto Not Present Gi2/0/8 xxxxx connected trunk a-full a-1000 1000BaseSX SFP Gi2/0/9 notconnect 6 auto auto Not Present Gi2/0/10 notconnect 6 auto auto Not Present Gi2/0/11 notconnect 6 auto auto Not Present Gi2/0/12 Uplink to 6509 connected trunk a-full a-1000 1000BaseLX SFP MDF# sh mod ^ % Invalid input detected at '^' marker.

MDF#sh module ^ % Invalid input detected at '^' marker.

Elliman-MDF#sh diag ^ % Invalid input detected at '^' marker.

MDF#sh span MDF#sh spanning-tree

No spanning tree instance exists.

6509 ====== cat6509#sh inter Gi2/15 status

Port Name Status Vlan Duplex Speed Type Gi2/15 DnLink MDF connected trunk full 1000

1000BaseLH cat6509#sh ha Cisco Internetwork Operating System Software IOS (tm) s72033_rp Software (s72033_rp-JK9O3SV-M), Version 12.2(17d)SXB6, RELEASE SOFTWARE (fc1) Technical Support: (c) 1986-2004 by cisco Systems, Inc. Compiled Fri 17-Dec-04 01:07 by hqluong Image text-base: 0x40020FBC, data-base: 0x422F0000

ROM: System Bootstrap, Version 12.2(14r)S9, RELEASE SOFTWARE (fc1) BOOTLDR: s72033_rp Software (s72033_rp-JK9O3SV-M), Version

12.2(17d)SXB6, RELEASE SOFTWARE (fc1)

cat6509 uptime is 18 hours, 1 minute Time since cat6509 switched to active is 18 hours, 0 minutes System returned to ROM by power-on (SP by power-on) System restarted at 16:43:40 EDS Sat Sep 15 2007 System image file is "sup-bootflash:s72033-jk9o3sv-mz.

122-17d.SXB6.bin"

This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption. Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

If you require further assistance please contact us by sending email to snipped-for-privacy@cisco.com.

cisco WS-C6509 (R7000) processor (revision 3.0) with 458752K/65536K bytes of memory. Processor board ID SAL0808UUZJ SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache Last reset from power-on Bridging software. X.25 software, Version 3.0.0. SuperLAT software (copyright 1990 by Meridian Technology Corp). TN3270 Emulation software.

2141 Virtual Ethernet/IEEE 802.3 interface(s) 116 Gigabit Ethernet/IEEE 802.3 interface(s) 1917K bytes of non-volatile configuration memory. 8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K). Standby is up Standby has 458752K/65536K bytes of memory.

Configuration register is 0x2102

cat6509# sh mod Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------

----------- 1 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC XXXXXXXXXXX 2 16 Pure SFM-mode 16 port 1000mb GBIC WS-X6816-GBIC XXXXXXXXXXX 3 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX XXXXXXXXXXX 4 24 CEF720 24 port 1000mb SFP WS-X6724-SFP XXXXXXXXXXX 5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE XXXXXXXXXXX 6 2 Supervisor Engine 720 (Warm) WS-SUP720-BASE XXXXXXXXXXX 7 8 Intrusion Detection System WS-SVC-IDSM-2 XXXXXXXXXXX

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------

------------ ------- 1 000e.38c4.3cb8 to 000e.38c4.3cc7 1.7 12.2(14r)S3

12.2(17d)SXB Ok 2 000e.3828.e7a0 to 000e.3828.e7af 1.7 12.2(14r)S3 12.2(17d)SXB Ok 3 000f.8f3e.fa98 to 000f.8f3e.fac7 1.5 12.2(14r)S5 12.2(17d)SXB Ok 4 0012.7f51.4504 to 0012.7f51.451b 2.1 12.2(14r)S5 12.2(17d)SXB Ok 5 000d.6536.32a8 to 000d.6536.32ab 3.1 7.7(1) 12.2(17d)SXB Ok 6 000d.bda3.e9b0 to 000d.bda3.e9b3 3.1 7.7(1) 12.2(17d)SXB Ok 7 0014.a9bc.f788 to 0014.a9bc.f78f 5.0 7.2(1) 6.0(2)E1 Ok

Mod Sub-Module Model Serial Hw Status

--- --------------------------- ------------------ ------------

------- ------- 1 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX

2.0 Ok 2 Distributed Forwarding Card WS-F6K-DFC3A XXXXXXXXXXX 2.0 Ok 3 Distributed Forwarding Card WS-F6700-DFC3A XXXXXXXXXXX 2.1 Ok 4 Distributed Forwarding Card WS-F6700-DFC3B XXXXXXXXXXX 2.0 Ok 5 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX 2.2 Ok 5 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX 2.2 Ok 6 Policy Feature Card 3 WS-F6K-PFC3A XXXXXXXXXXX 2.2 Ok 6 MSFC3 Daughterboard WS-SUP720 XXXXXXXXXXX 2.2 Ok 7 IDS 2 accelerator board WS-SVC-IDSUPG XXXXXXXXXXX 2.3 Ok

Mod Online Diag Status

--- ------------------- 1 Pass 2 Pass 3 Pass 4 Pass 5 Pass 6 Pass 7 Pass cat6509# sh diag

cat6509#sh span

No spanning tree instance exists.

cat6509#

The IOS software being used on the 6509 is quite dated. Not saying it is the cause of your issue but I would definitely start evaluating if an IOS upgrade should be considered. I know it can be a daunting task to pick a new IOS release for a 6500. I am running

12.2(18)SXF6 on a SUP720, however, it is a lab environment.

Cisco has yanked 3750 IOS images 12.2(25)SE and 12.2(25)SEA, it looks like you have 12.2(25)SEC.

I would upgrade ONE of the troubled 3750 switches to say 12.2(35)SE5 to see it it makes any difference

Question - if you move a troubled 3750 switch to a different port on the 6509 does the problem follow ?

another thing you could try is disabling DTP on the troubled trunks - BOTH ends

int type x/y switchport switchport trunk encapsulation dot1q switchport mode trunk switchport nonegotiate

OBTW does the 6509 need to be powered cycled oftern ?

If so what is the situation that requires the power cycling?

6509 doesn't need power cycling often. the room that its in, has some AC issues so it rebooted on its own last week and that is when we saw the switch failures (again). It happened a month ago too but that was a scheduled reboot of the router.

I guess I can try to turn DTP off, that won't hurt anything but if it is DTP then that would fail ALL switches and not just some and whats weird is that its the same set of switches all the time.

I did a side by side comparison of a working and a problem switch and found no diff. (other then IP) in config. both on switches and their trunk definition on 6509. I do have err-disabled recovery off on both

6509 and switches so I am going to see if I turn that on (recovery), does that make any diff?

Question: If the switch side gets 'err-disabled', would the 6509 side also show 'err-disabled'? or would it show disconnected. we tried shut/ no shut on 6509 when this happened and that never borough the connection up. I am wondering if that is the problem here.

Don't overlook the fact this this may be a bug. IOS code is sometimes crap.

Just curious - what are your thoughts behind STP being shut off? I'm assuming a star topology?

On your uplinks, have you played with the UDLD settings to see if there's any change?

Are the GBICs you bought directly from Cisco? If not, there are many, many counterfeits that will act flakey. They will not always throw an error in the syslog.

snipped-for-privacy@yahoo.com wrote:

I suggest do the following:

1. upgrade ONE to troubled 3750 to the IOS that I suggested.

1. disable DTP on the two or three of the trunsk to the the troubled

3750's

1. Move one of the troubled 3750 to a different 6509 port to see if problem follows.

Then schedule a reboot to observe what occurs or does not occur.

Capture the output of "show interface trunk"

OBTW I ALWAYS disable DTP on trunks as it is just one more thing that can go wrong. If a link is a trunk then it is a trunk ...

Now using 12.2(18)SXF11 as Cisco junked SXF, SXF1 thru SXF9