Have a question or want to start a discussion? Post it! No Registration Necessary. Now with pictures!
September 3, 2005, 7:05 am
rate this thread
HURRICANE KATRINA - UPDATE #18
The NRRT (Network Response Recovery Team) is monitoring the recovery
and restoration activities in the Gulf Region. Here is the latest
status on Hurricane Katrina from the Customer Relations/NSM Event
Subject - Hurricane Katrina Advisory # 18, (Please note all new
updates are in red.)
Date/Time – Friday, 13:00 CDT, September 2, 2005
Customer Impact -
Out of Service and impaired conditions on the wireline and wireless
networks as a result of the 2nd landfall of Katrina in the LA, MS, AL
Event Synopsis –
Sprint Nextel has a wireline switch site offline in New Orleans, a
loss of a POP in Biloxi, MS and multiple cell site towers offline due
to Hurricane Katrina.
Current Status –
Effective September 1st, the NRRT team will meet two times per day.
An update message to these distribution lists will be sent before
13:00 CDT each day unless conditions warrant otherwise. For
additional status updates please access.
EVN5 has been opened for tickets related to the second landfall for
Katrina, tracking 1851 tickets and EVN4 is related to SONET node
isolation and remains open to track 567 tickets. LEC1 has 9 tickets
and has TSP circuits.
General updates and statuses –
Business Service Assurance Center responsible for wireline service
assurance BSAC has a team set up to take internal calls. The
purpose is to prevent calls from Sales and Sales Support from
competing with the timely servicing of actual customer calls.
IP proactive tickets generation has been off.
The NRRT team is meeting via a conference bridge approximately two
times each day.
Sprint NEXTEL's Business Continuity Office, (BCO) is currently
monitoring storm, network, and employee impacts.
The CMC Scheduled Maintenance freeze for the following: New Orleans,
Baton Rouge, Shreveport, Montgomery, Birmingham, Jackson, Louisville,
and the Ft. Worth Distribution Center - in effect until further
The NETS team is also assisting with prioritization on all CDMA site
recovery and COW deployment efforts.
Current Network Impact
Please see wireline/wireless summary information below
The most significant network impacts:
The Sprint New Orleans wireline switch is offline. The site has
experienced water intrusion. The Switch, Digital Cross Connects, and
other network elements are offline. There are 600+ DS3s down,
multiple SONET ring nodes isolated, and switched access services from
numerous NPA/NXXs impaired. Sprint Nextel 1+ customers and Sprint
Local customers home out of New Orleans (Gulf States as far east as
the Panhandle and even parts of mid Florida are experiencing fast
busies and reorder tones). For dedicated access, customers homed to
or through this switch for ATM, IP, Frame Relay etc. are out of
service. Sprint has reviewed the facility assessment and has made
the decision to move forward with the re-homing of 13 of 14 POP’s
currently homed to New Orleans to other switch locations.
The Biloxi POP was damaged and non recoverable. Access Services
through and from this POP are out of service. Two SONET rings are
down. The replacement building is scheduled to be onsite September
The Kenner CDMA wireless switch is back up as of Wednesday
afternoon, August 31st. Sprint continues to experience loss of
visibility to the switch and SS7 availability. Local, roaming and LD
will fail intermittently until this is fully stabilized. We
understand 3G text messaging is working at this time.
Sprint Nextel has numerous cell towers out of service due to power,
T1 (or both) issues.
1+ Long distance calling from the local territories in the Panhandle
are impacted because of the New Orleans situation.
iDEN reports additional problems in the New Orleans area. Bell South
has lost a main Central Office due to generator failure. This has
increased loss of cell coverage in the region.
The situation in New Orleans continues to deteriorate with severe
flooding, looting, and rioting. Sprint has teams in the area and has
secured access into the city to make onsite evaluations. Efforts are
being hampered by the rapid changes. Security and safety has become
a critical concern for Sprint Nextel personnel in the region. As a
result further physical site restoration has been temporarily
suspended. Sprint Security is working to provide additional
The wireline Circuit Restoration team is fully activated and
prioritizing circuits along these general guidelines:
§ Critical Services
§ High priority business sites
Sprint’s Circuit Restoration Team, along with ASA, BSAC and NSM
organization are in the process of identifying TSP and other critical
services in the effort to prioritize restoration in the region for
all circuits, not just pass through.
o The first Restoration Priority list has been delivered to the
field and is posted on the BSAC web site. Priority is based on TSP,
Public Safety, ticket severity levels, and field input. The list
will be updated COB each day and will be used by BSAC and Network in
restoration. The opportunity to restore services is in three areas:
Circuits that traverse through New Orleans but do not originate or
terminate there Circuits homed to New Orleans that are not in areas
affected by the storm. For example the Panhandle of Florida. Re-home
efforts to other network elements like Frame, ATM, IP or Voice
switches have their own set of challenges with this effort. Circuits
that are in the affected areas that are in the impacted Gulf state
area will take the longest to restore. Karen Gordey, NSM Network
Notification Team, is the lead for the Legacy Sprint wireline and
wireless circuits. Account teams can send priority circuits to Karen
and include the trouble ticket number, circuit number, and
justification for prioritization. (The NETS organization is lead for
Legacy Nextel IDEN services, as noted above.) TSP restoral - 93
TSP’s, 56 restored, 37 non-workable. 248 critical circuits have been
identified as of this time. Progress has not been captured. These
circuits are largely non-TSP health and safety with a few extremely
large business circuits. Medium and Routine should be prioritized
next. No ETR can be given for circuit restoral at this time. Because
of the volume of requests being submitted for priority restoration,
new regional / segment POC’s will be established for submissions and
escalations. More details forthcoming later today. Trunk re-homing
for local territories long distances services to AT&T is reported to
have been finished and calls are reporting to be completing at this
time. We are in the process of augmenting long distance trunks
within the BellSouth territory with multiple IXC carriers. It is
anticipated that incremental relief should be felt on 1+ dialing in
the days ahead, Toll Free origination relief will not be felt until
new access routes are built. We have contacted approximately 2/3 of
the independent telephone operating companies on the issue of 1+
dialing. There have been varying degrees of response ranging from
the LEC already moving traffic to other carriers and / or performing
capacity analysis. The circuit restoration team is meeting hourly
for updates, they have assigned a project manager for each of the key
areas for restoration. Sales and Sales Support teams should not use
order activity to attempt to restore service. These resources are
receiving numerous inquiries which is taking away from their daily
responsibilities. Additionally this capacity needs to be reserved
for TSP and critical services.
Summary Network Impact Data
No new updates on Bell South at this time.
Sprint has lost visibility into multiple sites (monitoring/alarm
streaming) in the region including because of the Switch outage in
New Orleans Switch has been powered down to protect network
o 7 New Orleans DACS affecting 602 DS3 are out of service.
o 4 SONET ring node isolation
o 13 SONET rings switch failed
o WATS traffic enabled through IXC carriers, however some
impairment still exist .
o Numerous NPA’s and POP’s supported out of New Orleans.
o Physical testing of the fiber plant appears to be intact in New
Baton Rouge POP is on commercial power.
Hammond POP is on generator
Lafayette POP is on generator, no monitoring visibility
Kentwood REGEN is on generator, no monitoring visibility
Krotz Springs REGEN is on generator, no monitoring visibility
Mobile POP is on generator
Montevallo REGEN is on generator
Biloxi POP / Regen has been declared destroyed.
o 2 Sonet rings down affecting 32 DS3’s 4 SONET rings successfully
Jackson POP is on commercial power.
Brookhaven REGEN is on generator
Wireless CDMA Network:
Sprint has lost command and control (monitoring, alarms, etc.) for
many of the CDMA access facilities in the impacted area.
There are 324 sites down across the affected area with 182 Sprint
and 142 partners.
Wireless iDEN Network:
There are 204 sites down across the affected area with 152 iDEN and
Next Status Update - By 13:00 CDT Saturday, September 3rd.
- » 'Human Error' caused two-thirds of Minnesota 911 calls to be dropped, outage...
- — The site's Newest Thread. Posted in » General Telecommunications Forum