We are a west-coast (San Francisco) firm. Our website, which is hosted by Verio, has been down since at least this morning. The problem appears to be a "proactive shutdown" by Verio's data center in Boca Raton, Florida. Since Verio provides hosting services for millions of customers, I expect that this action in Boca Raton is having internet-wide implications and affecting many, many businesses.
The Verio website shows the following status messages:
10/24/2005 7:00 pm ET updateTo Verio shared hosting customers:
Verio is continuing to work with the building/facilities management on an on-site technical assessment of the generator system. Our team will be at work throughout the evening as needed to continue work on power and service restoration. A new status update will be posted here at
8:00 pm ET.Many of our shared hosting customers have been impacted by Hurricane Wilma's effects in the south Florida area. Our Boca Raton data center facility did experience power loss this morning, and backup generators, which were tested just four days ago and found to be working properly, are not operational at this time. At this time, we have proactively powered down servers to minimize impact to our shared servers located in the Boca Raton data center. We will restore these servers once power comes back online. In the meantime, we are aggressively working with the local power company to assess when power will be restored, and we are working with the building facilities management to restore service to backup generators as well. We will continue to provide updates and any time estimates on this site as we have them.
Thank you for your patience during this time. We understand the impact of this action to your business, and the Verio Disaster Recovery team is diligently working to have power and service restored as quickly as possible.
----------------------
Our website came up sometime this morning. The following message was posted on Verio's status page:
10/25/2005 10:00 am ET updateTo Verio's domestic and international hosting and colocation customers impacted by Hurricane Wilma and the Florida data center service interruption:
Our Boca Raton facility is currently operational on backup generator power with services in the process of being restored. All servers either have already been powered on or are in the process of coming online. We are continuing to monitor any issues during this process in an effort to minimize impact to the servers and data. We are also continuing to monitor the restoration of primary power service to minimize any impact should the facility transfer over from generator power to primary municipal power automatically. At this time, that is a possibility and one we are watching closely due to possible service interruption.
Due to weather conditions related to Hurricane Wilma, our south Florida datacenter lost primary and backup power at approximately
9:40am ET, Monday October 24th. Verio's other datacenters, including premier datacenters located throughout the US, Europe, and Asia, were not impacted by this outage.At approximately 8:40am ET on October 24th, Verio lost municipal power to the south Florida facility. The facility has a system of five secondary generators which supply power in the event of a power outage. The secondary backup system began providing power when municipal power was lost. When issues with the backup system became apparent, the system was immediately converted to battery backup and systems operated for as long as possible. After battery backups were depleted, Verio electively turned off systems in the datacenter to minimize impact to servers and data.
Due to extremely difficult weather conditions outside the facility, technicians were unable to effectively access the backup generators for much of the morning and early afternoon. As technicians were able to access the system, they began doing an initial assessment, and the generator vendor was also able to dispatch technicians to assist with repairs. Due to hazardous conditions, including high winds, power lines, and debris, the vendor technicians arrived on-site during the early evening hours, October 24th.
After diagnosing the problem and effecting the appropriate remedy, the generators were sequentially brought back on line starting at approximately 1:40 am ET on October 25th. Once the generator plant was producing the correct voltage and frequency with the requisite stability, power was switched to the data center's split-system (DX) computer room air conditioning (CRAC) units, and simultaneously to the chilled water plant that feeds the bulk of the facility's cooling infrastructure. Once the cooling system restored safe operating temperatures in the raised floor areas, Verio was able to energize the data center network equipment, before proceeding to begin powering up the shared hosting and colocation environments at approximately 7:10 am am ET on October 25th.
In preparation for Hurricane Wilma, a full battery of tests was conducted on the generators and systems. These tests were last conducted on October 19th, just days prior to Hurricane Wilma passing over our facility. These tests all came back positive, with no issues. We are continuing to fully investigate these issues in our follow up efforts.
Robert Weller