A message to our American players

[Pedro-NF]

I believe I speak for all players of the [>NF<] Server when I say that I wish the best of luck to our American players and friends living in the area that will be hit by the Sandy hurricane. Our hearts and minds are with you guys!

The Sacred Emperor

[Pedro-NF]

Upcoming routine InterNAP maintenance late 11/1

Oct 29 2012 11:56:55 AM PT

InterNAP is planning routine maintenance to reboot one of their internal routers between 11pm EDT on 11/1 and 4am EDT on 11/2. This may result in short bursts of packet loss and/or higher-than-normal latencies for your service at this location.

In an unrelated matter, InterNAP is continuing to monitor the situation in regards to hurricane Sandy. Our equipment is several floors up in the facility and the facility has not been affected by previous storms (it is well-designed and well-protected, with significant backups available), so we do not anticipate an interruption in service to occur.

[Pedro-NF]

Datacenter flooding

Oct 29 2012 08:27:43 PM PT

InterNAP has sent us the following notification:

Please be advised that the NYM-EXT1 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.

The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.

We hope that flood waters will recede enough that generators can function and/or that utility power can be restored before this happens, but we can't tell if that will be the case.

If a complete power outage takes place, our managed services should not be significantly affected beyond the downtime. This is because all are configured to come back online properly when power is restored, and managed game servers (with the exception of Minecraft servers, which are snapshotted hourly) rarely write to disk, meaning that disk corruption is unlikely. However, unmanaged services would also go down. If you run an unmanaged VDS or machine, we recommend that you back up critical files (such as databases) that might be writing to disk at the time of any later abrupt shutdown, and that you consider stopping those services in advance to be safe.

[Pedro-NF]

We are officially down. No predictions on when we'll be back online.

[JR^^]

75 broad st right near battery park much of that area is underwater.

not much happened where i am some wind and rain.
but i am upstate in NY a ways.
we had a wind shear event where i live a few years ago and it sounded like a freight train and knocked
down or damaged 90% of the trees in the area but it was very quick not like this storm.

[Pedro-NF]

NYC location remains offline

Oct 30 2012 05:28:52 PM PT

Our NYC location remains offline and InterNAP is working to bring fuel to the generators in order to turn back up the power. They are declining to provide any sort of ballpark ETR at this time.

[Pedro-NF]

I believe I speak for all players of the [>NF<] Server when I say that I wish the best of luck to our American players and friends living in the area that will be hit by the Sandy hurricane. Our hearts and minds are with you guys!

Idk somehow I would have never expected such post from you, Pedro.

Are you going to finally pledge allegiance to The Sacred Emperor now?

[Pedro-NF]

NYC power restored

Oct 31 2012 12:36:37 AM PT

InterNAP has related to us the following:

The generators have been refueled, and engineers have successfully restored power to the site via generator. Please begin checking your systems now. There is still no customer access being granted at this time so please do not make plans to visit the data center until you have been notified by Internap that building access has been restored.

Followed by:

Subsequent to power returning via generator as previously notified, the power was lost again briefly. The onsite engineers are investigating and work to stabilize power is ongoing. The NOC will continue to update customers as the work continues, and confirm once Remote Hands services are available.

Followed by:

As previously notified, the NYM-EXT1 facility has had generator power restored, and engineers have continued to work to stabilize both the generator and additional site infrastructure in support of our customers.

At this time site engineers have been able to confirm that the generator is running normally, and is at full fuel capacity. The brief drop in generator that took place has been fully investigated and it is not expected to be a concern moving forward. Additionally site engineers have returned the UPS systems to service and confirmed that the UPS' are operating normally and helping to support critical loads. Site engineers have also returned all CRAC units to service at this time. The local site engineers believe that the generator and UPS systems are operating in a stable fashion at this time and as noted, we have ample fuel on site and are coordinating additional fuel delivery.

From what they have said, we don't expect further power outages. However, some of InterNAP's upstreams in NYC are still offline and may go up and down, so you may see higher-than-normal pings or slight packet loss at times as InterNAP has to route around them and compensate.

After the last notification, we worked to systematically bring all services back online. All but one of our game server machines are online right now (the customers on the machine that is offline are being contacted separately). Voice servers are also working normally, and VDSes should all be online. If you have a unmanaged dedicated machine of your own at this location and you are not able to connect to it, please contact us ASAP so that we can pull up its console and investigate.

We greatly appreciate your patience as we and InterNAP dealt with the damage from this powerful storm! Total downtime for most services was a bit under 15 hours (11:48am EDT through 2:15am EDT). If you require a day's SLA credit for the downtime, please let us know, and we will be happy to issue this.

[Pedro-NF]

New NYC connectivity loss (but not loss of power)

Oct 31 2012 03:37:22 AM PT

We lost contact with all our equipment in NYC again at approximately 5:48am EDT this morning (Oct. 31). InterNAP has not sent out an official announcement yet about this problem, but their NOC answered our telephone inquiry by stating that one of InterNAP's other facilities in NYC is without power, and as a result, our facility does not have connectivity. They tell us that the generators are still working properly for us, which means that your service is continuing to run in the background and should simply pop back online when they get the link fixed, continuing as it was at the start of the outage.

[Pedro-NF]

THE SERVER IS BACK ONLINE!!!

[Pedro-NF]

Lol

Is that a yes?

[Pedro-NF]

That's a lol.[/color]

Lord, O Lord...?