Work continues throughout our city to restore electrical service as does our work to restore generator operation. We have an ETA of the generator of 1700 central time from the machine shop that will fabricate replacement parts. We are monitoring the condition of the electrical utility power closely. We detected erroneously out-of-spec power on one phase yesterday, and we isolated all client machines from the electrical utility to protect your equipment. It's bad enough that you're offline, but we will not take any chance of damage to your equipment. So electrical utility power can not be utilized until the utility is supplying it safely. The utility, Evergy Energy has not provided an ETA at this time.
As soon as one of (1) the electrical utility or (2) our generator is restored, your service will resume automatically. We will review every server of every client after electrical service is back online to make sure your systems are responding. PLEASE make a note in your ticket now if your servers are configured to never respond to network traffic, or if they do not listen to standard ports. If you tell us in your ticket how to verify your server is responding as you expect, this will expedite our response, which we very much intend to do.
Also we would like to provide the following in case your clients may be asking:
- No. We have not been hacked. There is no known security breach at this time. We are keeping personnel on-site to physically secure your hardware while we await restoration of power.
- No. Your data is not lost. It is inaccessible and we are working to restore it ASAP. We will provide KVMoIP and IPMI access where possible after restoration of power, to give you control of any recovery process if you need it.
- Natural Disaster was the cause of this outage. Tornadoes are not uncommon in our area, but the disaster yesterday was "straight line wind" which has never been this powerful in our city. It has damaged much of the central continental United States.
- No employees were hurt during the disaster or response.
Greetings UnReal friends, it has been an unreal day indeed!
Around 1800 Central time Wednesday, December 15th, 2021, catastrophic wind storms struck our home in the Kansas City area. Wind as fast as 95 miles (150 km) per hour has leveled buildings, thrown trucks through buildings, and even blown 15 tonne generators over. Many have died. Please keep our city in your thoughts and prayers as families mourn.
Over 200,000 electrical customers are without service this evening with one of our power companies, in a city of only 250,000 homes.
UnReal Servers has also experienced damage to our generators as you are have no doubt noticed.
None of our employees were injured during this storm or our response to it, and we're going to keep it that way.
At 1935 Central time, electrical utility power was lost to our building and we went on generator. All staff began in transit to the datacenter.
Repair technicians have been working since 2000 central time in the cold and in the dark, on restoring generator service, and on restoring electrical utility (grid) power to our building, and also to the wider area. Hospitals are receiving priority by the utility.
Thank you for your continued patience as we work through this disaster. Instead of responding to individual tickets, we will be focusing our time to recover service as soon as possible. So stay tuned to these posts for updates.
Thursday, December 16, 2021