How to troubleshoot this network issue

Router log shows ( 10.10.10.2 is the peer router ):

Apr 14 04:05:06.592 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:06.812 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:22.405 HKT: %OSPF-4-NONEIGHBOR: Received database description from unknown neighbor 10.10.10.2 Apr 14 04:05:26.841 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from LOADING to FULL, Loading Done Apr 14 04:05:29.725 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done Apr 15 04:08:51.926 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.539 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.780 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp

40 state Standby -> Active Apr 15 04:08:53.028 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Standby -> Active Apr 15 04:08:53.357 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp 40 state Active -> Speak Apr 15 04:08:53.413 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Active -> Speak Apr 15 04:09:21.771 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:09:21.779 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:10:00.025 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from LOADING to FULL, Loading Done Apr 15 04:10:07.530 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done

Anybody finds any hint about this issue ?

Thanks so much

Reply to
bensonlei
Loading thread data ...

Router issue ( 10.10.10.2 is the peer router) :

Apr 14 04:05:06.592 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:06.812 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:22.405 HKT: %OSPF-4-NONEIGHBOR: Received database description from unknown neighbor 10.10.10.2 Apr 14 04:05:26.841 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from LOADING to FULL, Loading Done Apr 14 04:05:29.725 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done Apr 15 04:08:51.926 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.539 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.780 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp

40 state Standby -> Active Apr 15 04:08:53.028 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Standby -> Active Apr 15 04:08:53.357 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp 40 state Active -> Speak Apr 15 04:08:53.413 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Active -> Speak Apr 15 04:09:21.771 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:09:21.779 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:10:00.025 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from LOADING to FULL, Loading Done Apr 15 04:10:07.530 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done

Anybody finds the issue ?

Thanks so much for your kind help

Reply to
bensonlei

Router issue ( 10.10.10.2 is the peer router) :

Apr 14 04:05:06.592 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:06.812 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:21.961 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from EXSTART to DOWN, Neighbor Down: Dead timer expired Apr 14 04:05:22.405 HKT: %OSPF-4-NONEIGHBOR: Received database description from unknown neighbor 10.10.10.2 Apr 14 04:05:26.841 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from LOADING to FULL, Loading Done Apr 14 04:05:29.725 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done Apr 15 04:08:51.926 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.340 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.539 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from FULL to DOWN, Neighbor Down: Dead timer expired Apr 15 04:08:52.780 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp

40 state Standby -> Active Apr 15 04:08:53.028 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Standby -> Active Apr 15 04:08:53.357 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.340 Grp 40 state Active -> Speak Apr 15 04:08:53.413 HKT: %HSRP-5-STATECHANGE: FastEthernet0/0.301 Grp 1 state Active -> Speak Apr 15 04:09:21.771 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:09:21.779 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from EXCHANGE to DOWN, Neighbor Down: Dead timer expired Apr 15 04:10:00.025 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.333 from LOADING to FULL, Loading Done Apr 15 04:10:07.530 HKT: %OSPF-5-ADJCHG: Process 10, Nbr 10.10.10.2 on FastEthernet0/0.301 from LOADING to FULL, Loading Done

Anybody finds the issue ?

Thanks so much for your kind help

Reply to
bensonlei

It could be so many different things, the logs you post are the end result, not the cause.

Maybe you are dropping packets? Maybe the line goes down? Maybe your packets are scrambled? Maybe you have an MTU problem?

Look at the error counters on the interfaces in question on both sides. Watch the interface status. Try running network traffic pattern tests and throughput tests. Make sure your packets get through unscrambled.

Reply to
Doug McIntyre

Thanks so much for your suggestion.

No CRC error for the interfaces so far, but only the overrun errors. And it should not be the MTU problem due to more than 10 sites are the same configuration without such problem. what else problems left ? scrambled packets --- any suggestion to test this issue ?

Reply to
bensonlei

1st event is OSPF timeout - you have lost contact with the neighbour router over a vlan on Fa 0/0

and another to the same neighbour, but on a different vlan over the same port.

the OSPF adjacencies starting coming up but didnt make it (which seems a bit wierd and there dont seem to be all the log entries there should be)

spanning tree or something interrupted the link after it bounced? - note if we knew what your ospf settings looked like we might understand what is happening better

both back up

and down again

you have an ethernet link between 2 routers, and you are seeing either the neighbour router having problems, or the link between the 2 boxes.

Reply to
Stephen

=A5=CE=A4=E5=A6r -

Hi Stephen Thanks for the idea.

Over than 10 sites have the same configuration, but only this site has such problem. The network supports mission critical system, so we set up the IPsec tunnel between HQ and each site ( so the MTU =3D 1476 ); and also the hello-interval =3D 2 seconds; HSRP hello-interval =3D default...; we also support QoS for applications

Any hint to find out the root cause ?

Reply to
bensonlei

figuring out a log is a lot easier if you know the context.

i dont think anyone can work out much more until you explain how your network is laid out and what is there

Reply to
Stephen

Cabling-Design.com Forums website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.