The Server North
N

'blog

Home Main Site NOC/Info Site Twitter Stream Archives Login
You are currently viewing archive for March 2010
Mar
18
Blip warning. While we're not supposed to be affected by the UPS maintenance going on (and the mitigation work that started a few minutes ago), it's entirely possible that some of our suppliers and upstreams might be... but we've received no warning. Hopefully we don't feel anything.

Posted by: Myke |

Mar
17
All Clear. I want to mention that we're now on a different UPS, so we're unaffected by the breaker-repairs that're happening tomorrow, and we're off commercial power and protected again. I'll try to post a post-mortem tomorrow... it's been a hectic 2 days.

Posted by: Myke |

Mar
17
Approaching Normal. All services should be fully operational again. We have a minor problem with our monitoring, but that only affects accounting ;)

Please send a note to Support@ServerNorth.net or give us a call if you've got any problems. I (Myke) will stay on-site for another hour and a bit to make sure the power is stable (it should be!) and then I'll pack up. Only to do this again on April 17th, Diesel Day... but you didn't hear that from me.

Posted by: Myke |

Mar
17
Power is on. Restoring services now. Things went well.

Posted by: Myke |

Mar
17
Power is off. New circuit being 'converted' into place now.

Posted by: Myke |

Mar
17
One hour warning. All the new wiring's roughed in. Starting auditing and moving things with redundant power-supplies/inputs now.

Posted by: Myke |

Mar
17
EMERGENCY MAINTENANCE - 10PM Tonight All Toronto-based services will be disrupted around 10PM EDT (tonight) while we mitigate/move things around from one circuit to another to avoid power problems/more outages.

We're still auditing what will be affected, but we know for sure at least one core router and one switch will hit, so there will definitely be a brief outage while they're moved or manually 'failed(v)-over', otherwise many servers will be down for what we expect to be 10-30 minutes, but possibly as much as an hour - if things don't go well.

Expect another post here before 10PM... after that a bit of live tweeting.

Posted by: Myke |

Mar
17
Official notice At approximately 03:45 AM during the scheduled maintenance window for 602.UPS.3 the unit was to be returned from commercial power upon completion of its battery replacement and preventative maintenance. Upon transfer the input breaker experienced a mechanical failure. This has not caused any outage, however, the unit will need to remain in bypass and on commercial power until further notice. We are currently working with Powerware to determine the next course of action in resolving this issue. As of now the commercial power is stable, although any disruption in commercial power may cause a loss to the data center floor.



We will provide an update as it becomes available.



**Please extend the current maintenance window until further notice**

Posted by: Myke |

Mar
17
Emergency Power Shutdown Tonight, all power to our gear in Toronto will be shut down as the UPS maintenance did not go well, we're stuck unprotected on utility power until a circuit breaker is replaced.

We do not know the time yet, but should in the next few hours. We will be contacting all colocation customers directly. Hosting and Highspeed will also be affected, although we're trying to negotiate a few things, and we may be able to get a circuit to a different UPS installed so we'll only have blips as we move things, rather than suffer the full 45+ minute outage.

More soon.

Posted by: Myke |

Mar
11
Power Maintenance This upcoming Tuesday, 2010-03-16 at 23:00 to 2010-03-17 04:00, the UPS we're connected to in the Toronto datacenter will have it's batteries replaced. This isn't supposed to cause any outages, but you never know. Myke will be on-site (or very close by) if there's any problems. Hosting, Colocated servers, Highspeed (ADSL) and all other resources in Toronto may be impacted. We're also hoping to do a router upgrade, but it's a back-up unit, so there shouldn't be any reverberations from that.

Posted by: Myke |

Powered by NucleusCMS | Ported by VinhBoy | Designed by DemusDesign