Katrina, internal report / carrier

Have a question or want to start a discussion? Post it! No Registration Necessary.  Now with pictures!


  The NRRT (Network Response Recovery Team) is monitoring the recovery
and restoration activities in the Gulf Region.  Here is the latest
status on Hurricane Katrina from the Customer Relations/NSM Event
Notification team.

  Subject - Hurricane Katrina Advisory # 18, (Please note all new
updates are in red.)

  Date/Time – Friday, 13:00 CDT, September 2, 2005

  Customer Impact -

  Out of Service and impaired conditions on the wireline and wireless
networks as a result of the 2nd landfall of Katrina in the LA, MS, AL
Gulf Coast.

  Event Synopsis –

  Sprint Nextel has a wireline switch site offline in New Orleans, a
loss of a POP in Biloxi, MS and multiple cell site towers offline due
to Hurricane Katrina.

  Current Status –

 Effective September 1st, the NRRT team will meet two times per  day.
An update message to these distribution lists will be sent before
13:00 CDT each day unless conditions warrant otherwise.  For
additional status updates please access.

  EVN5 has been opened for tickets related to the second landfall for
Katrina, tracking 1851 tickets and EVN4 is related to SONET node
isolation and remains open to track 567 tickets. LEC1 has 9 tickets
and has TSP circuits.

  General updates and statuses –

 Business Service Assurance Center responsible  for  wireline service
assurance  BSAC has a team set up to take internal  calls.  The
purpose is to prevent calls from  Sales and Sales Support from
competing with  the timely servicing of actual customer  calls.

 IP proactive tickets generation has been off.

 The NRRT team is meeting via a conference  bridge  approximately two
times each day.

 Sprint NEXTEL's Business Continuity Office,  (BCO)  is currently
monitoring storm, network, and employee  impacts.

 The CMC Scheduled Maintenance freeze for the  following: New Orleans,
Baton Rouge, Shreveport,  Montgomery, Birmingham, Jackson, Louisville,
and the Ft.  Worth Distribution Center - in effect until further

 The NETS team is also assisting with prioritization on all CDMA  site
recovery and COW deployment efforts.

  Current Network Impact

  Please see wireline/wireless summary information below

  The most significant network impacts:

 The Sprint New Orleans wireline switch is  offline.  The site has
experienced water intrusion.  The  Switch, Digital Cross Connects, and
other network  elements are offline.  There are 600+ DS3s down,
multiple  SONET ring nodes isolated, and switched access services from
numerous NPA/NXXs impaired.  Sprint Nextel 1+  customers and Sprint
Local customers home out of New  Orleans (Gulf States as far east as
the Panhandle and  even parts of mid Florida are experiencing fast
busies  and reorder tones).  For dedicated access, customers homed to
or through this switch for ATM, IP, Frame Relay  etc. are out of
service.  Sprint has reviewed the  facility assessment and has made
the decision to move  forward with the re-homing of 13 of 14 POP’s
currently  homed to New Orleans to other switch locations.

 The Biloxi POP was damaged and non  recoverable.  Access Services
through and from this POP are out of  service.  Two SONET rings are
down. The replacement  building is scheduled to be onsite September

 The Kenner CDMA wireless switch is back up as  of  Wednesday
afternoon, August 31st. Sprint continues to  experience loss of
visibility to the switch and SS7  availability.  Local, roaming and LD
will fail  intermittently until this is fully stabilized.  We
understand 3G text messaging is working at this time.

 Sprint Nextel has numerous cell towers out of  service due to power,
T1 (or both) issues.

 1+ Long distance calling from the local  territories in the Panhandle
are impacted because of the  New Orleans situation.

 iDEN reports additional problems in the New Orleans area.  Bell South
has lost a main Central Office due to generator failure.  This has
increased loss of cell coverage in the region.

  Restoration Efforts

 The situation in New Orleans continues to  deteriorate with severe
flooding, looting, and rioting.  Sprint has teams in the area and has
secured access into  the city to make onsite evaluations. Efforts are
being  hampered by the rapid changes.  Security and safety has  become
a critical concern for Sprint Nextel personnel in  the region.  As a
result further physical site  restoration has been temporarily
suspended.  Sprint  Security is working to provide additional

 The wireline Circuit Restoration team is fully  activated and
prioritizing circuits along these general  guidelines:

  §  TSP

  §  Critical Services

  §  High priority business sites

 Sprint’s Circuit Restoration Team, along with  ASA, BSAC and NSM
organization are in the process of  identifying TSP and other critical
services in the effort  to prioritize restoration in the region for
all circuits,  not just pass through.

  o  The first Restoration Priority list has been  delivered to the
field and is posted on the BSAC web  site.  Priority is based on TSP,
Public Safety, ticket  severity levels, and field input.  The list
will be  updated COB each day and will be used by BSAC and Network  in
restoration. The opportunity to restore services is in  three areas:
Circuits that traverse through New  Orleans but do not originate or
terminate there  Circuits homed to New Orleans that are  not in areas
affected by the storm.  For example the Panhandle of Florida.  Re-home
efforts to other network  elements like Frame, ATM, IP or Voice
switches have their own set of  challenges with this effort.  Circuits
that are in the affected areas  that are in the impacted Gulf state
area will take the longest to restore.  Karen Gordey, NSM Network
Notification  Team, is the lead for the Legacy Sprint  wireline and
wireless circuits.  Account  teams can send priority circuits to Karen
and include the trouble ticket  number, circuit number, and
justification for prioritization.  (The  NETS organization is lead for
Legacy  Nextel IDEN services, as noted  above.)  TSP restoral - 93
TSP’s, 56 restored,  37 non-workable.  248 critical circuits have been
identified as of this time.  Progress  has not been captured.  These
circuits  are largely non-TSP health and safety  with a few extremely
large business  circuits.  Medium and Routine should be  prioritized
next. No ETR can be given  for circuit restoral at this time.  Because
of the volume of requests being  submitted for priority restoration,
new  regional / segment POC’s will be  established for submissions and
escalations.  More details forthcoming  later today.  Trunk re-homing
for local territories  long distances services to AT&T is  reported to
have been finished and  calls are reporting to be completing at  this
time.  We are in the process of augmenting  long distance trunks
within the  BellSouth territory with multiple IXC  carriers.  It is
anticipated that  incremental relief should be felt on  1+  dialing in
the days ahead, Toll Free  origination relief will not be felt  until
new access routes are built.  We have contacted approximately 2/3 of
the independent telephone operating  companies on the issue of 1+
dialing.  There have been varying degrees of  response ranging from
the LEC already  moving traffic to other carriers and /  or performing
capacity analysis.  The circuit restoration team is meeting  hourly
for updates, they have assigned  a project manager for each of the key
areas for restoration.  Sales and Sales Support teams should  not use
order activity to attempt to  restore service.  These resources are
receiving numerous inquiries which is  taking away from their daily
responsibilities.  Additionally this  capacity needs to be reserved
for TSP  and critical services.

  Summary Network Impact Data

  No new updates on Bell South at this time.

  Wireline Network:

  Sprint has lost visibility into multiple sites  (monitoring/alarm
streaming) in the region including  because of the Switch outage in
New Orleans.


 New Orleans Switch has been powered down to  protect network

  o  7 New Orleans DACS affecting  602  DS3 are out of service.

  o  4 SONET ring node isolation

  o  13 SONET rings switch failed

  o  WATS traffic enabled through IXC  carriers, however some
impairment still exist  .

  o  Numerous NPA’s and POP’s supported  out of New Orleans.

  o  Physical testing of the fiber plant appears to be intact in New

 Baton Rouge POP is on commercial power.

 Hammond POP is on generator

 Lafayette POP is on generator, no monitoring  visibility

 Kentwood REGEN is on generator, no monitoring  visibility

 Krotz Springs REGEN is on generator, no  monitoring visibility


 Mobile POP is on generator

 Montevallo REGEN is on generator


 Biloxi POP / Regen has been declared  destroyed.

  o  2 Sonet rings down affecting 32 DS3’s  4 SONET rings successfully

 Jackson POP is on commercial power.

 Brookhaven REGEN is on generator

  Wireless CDMA Network:

  Sprint has lost command and control (monitoring, alarms,  etc.)  for
many of the CDMA access facilities in the impacted area.

  There are 324 sites down across the affected area with 182  Sprint
and 142 partners.

  Wireless iDEN Network:

  There are 204 sites down across the affected area with 152 iDEN and
52 partners.

  Next Status Update - By 13:00 CDT Saturday, September 3rd.


Site Timeline