The Server North
N

'blog

Home Main Site NOC/Info Site Twitter Stream Archives Login
You are currently viewing archive for March 2012
Mar
15
Diesel Day 2012-04-14 (Toronto Datacenter) As per previous years, 151 Front Street West, Toronto will be performing annual power maintenance, where all incoming power from Hydro to the building is shut down from 6AM till 6PM on Saturday April 14th, 2012.

Hence the name, all of our and client's colocated equipment will be powered by on-site diesel generators. This is a period with high risk of outage and power-failures, especially considering our surprise move to a new UPS last week.

We will be performing extensive maintenance & upgrades on our own equipment, which means there will be interruptions in many services throughout the day. We're looking to do OS upgrades on the SAN, platform updates & memory upgrades to the VPS cluster, and time permitting - possible restructuring of our DSL routers.

There's a possibility we will be transferred back to the new, permanent UPS that replaced the ailing one, but we don't yet have a time-table for that. If possible - we're try to do that at the same time.

Posted by: Myke |

Mar
09
Disempowerating A view from one of our PDUs:

Posted by: Myke |

Mar
09
ALL CLEAR Everything should be up & running normally. If you have any issues, please get in touch with us via the usual channels.

Posted by: Myke |

Mar
08
Almost All Clear We're having a minor issue with some legacy jails right now... primarily the one that has our website (whodathunkit?)

Otherwise, we're just about done and will be working on re-enabling all the safe-guards that we turned off for the change-over.

Posted by: Myke |

Mar
08
Most services back online There's some Jail VPSes and little bits of routing that still need to be tweaked, but otherwise we're almost done! :)

Posted by: Myke |

Mar
08
Power is back on DSLs and Colo'd servers should be coming back online. Webhosting and VPS will take longer.

Posted by: Myke |

Mar
08
Shutdown has started. Please do not adjust your sets.

Posted by: Myke |

Mar
08
E-Mail sent to customers re: Outage Tonight Due to a failing power-system in our Toronto Datacenter, we will be shutting down ALL Toronto-based services at 11PM tonight to transition to a temporary UPS.

This includes:
- ADSL Internet (PPPoE & HSA)
- Web-hosting
- Virtual Private Servers (VPS/Jail/OpenVZ/KVM)
- Colocated & Dedicated Servers
- transit links and interconnections

If you are receiving this email, you are affected. If you are receiving this email, you are affected.
If you are receiving this email, you are affected. If you are receiving this email, you are affected.
If you are receiving this email, you are affected. If you are receiving this email, you are affected.
If you are receiving this email, you are affected. If you are receiving this email, you are affected.

We will be very busy at 11PM working to get services back online as quickly as possible. If you have the means, please follow along on our 'blog at http://status.servernorth.net/ and for even more high-paced updates, our Twitter stream is here: https://twitter.com/ServerNorth

If you have colocated servers or Virtual servers, please begin shutting them down around 10:30PM.

We hope to have all services up as quickly as possible, but as we are cold-starting most of our network, there are sequences and certain adjustments that need to be made.

Our cable Internet (via Rogers & Cogeco), enterprise/IMAP E-Mail services and Montréal facility will not be shutdown, however some things may affected as annexed-to-Toronto services such as (reverse) DNS are briefly unavailable.

We apologize for the short notice, however we weren't given much warning either. Unfortunately this task will need to be repeated in the coming weeks to return to regular power systems, but we do not yet have a date - and there will be more lead time.

Overall, we hope to have everything restored within the hour, and we will narrate on Twitter as we go.

Best regards,

Mike Geiger & The ServerNorth.net Crew

PS: Fundamentally this means we're unplugging the power cords from one outlet to another - 20 seconds of work, but it's all the shutdown & start-up sequences that will take all the time.

Posted by: Myke |

Mar
07
POWER MAINTENANCE 2012-03-08 23:00 Our Toronto colocation facility will be offline from 11PM Thursday night for a power-supply change. The outage/impact will be varying in length, we hope to have all services restored within an hour.

The order of turn-up is:
  1. Core Infrasctructure & Uplinks
  2. Access Networks
  3. Customer Colocation
  4. SAN
  5. Virtual Private Servers
  6. Webhosting

Posted by: Myke |

Mar
03
Power Changes/Disruptions in Toronto This Week In the last hour we've received notice from facilities that one of the UPSes powering our rack is failing and that all loads need to be shifted to a different unit. This change-over will have to be done before the 9th.

We haven't scheduled the change-over yet, but it will be sometime between Wednesday and Friday.

Watch this space for updates. This will disrupt all Toronto facility services, although we expect to make network, VPS and other transitions as brief as possible.

It's expected this change will need to be reverted in the coming weeks as well.

(Incidentally, we've heard rumour that the annual building power maintenance will be April 14th, we plan to do several upgrades & maintenance that day as well.)

Posted by: Myke |

Powered by NucleusCMS | Ported by VinhBoy | Designed by DemusDesign