The Server North
N

'blog

Home Main Site NOC/Info Site Twitter Stream Archives Login
You are currently viewing archive for June 2010
Jun
23
ADSLs coming back up We're not asking questions... yet.

Posted by: Myke |

Jun
23
Massive ADSL Outage It looks like it might be a Bell issue... we're making calls...

An upstream router has outright failed. The redundant/failover system appears not to have worked. People are working on it. There are tens-of-thousands down presently.

To itemize the past 24 hours:
Jun
23
Your Reaction to the Earthquake Observe the moment the 5.x Earthquake hit near Ottawa, Ontario:

Uplink Graph


This is just one of our 4 main uplinks, the others show similar spikes, except the Ottawa Internet Exchange, where we saw a clear drop in traffic.

So despite a pile of ADSL highspeed users getting knocked offline, from what we assume are broad power failures & lines down, there's a clear spike right as the ground shook.

Posted by: Myke |

Jun
23
Earthquake We're okay. Looks like all our gear is okay. Are you? We saw a lot of users go down...

Rogers seems to not be working (our phones are with them) so if you need help or something, call Myke's Telus phone at 613-223-6580

Posted by: Myke |

Jun
22
iOS4 (iPhone OS) vs. Mail It's painfully apparent that there's some incompatibility or bug in the new Apple iPhone Operating System that's causing rampant crashing with Mail.app on upgraded phones & Touches. We're aware of the problem and do not yet have a work around. The developer of our server software is working on the problem and we hope to have a fix deployed by the weekend.

Until then, if mobile E-Mail is important to you, hold off on upgrading your phone.
One can always hope Apple with fix the problem on their end too...

Posted by: Myke |

Jun
14
Yup, done. Going home now. Thanks for all the packets. :)

Posted by: Myke |

Jun
14
Done?! Aside from some internal/management stuff, we're done!

Posted by: Myke |

Jun
14
Network Blip Impending at 2PM We'll be finishing up the hot-cuts with a bang at 2PM. Expected that ADSL connections will reset, but that should be the worst of it. We've already finished doing half of the other changes, FYI.

Posted by: Myke |

Jun
14
Hot-Cuts happening today We'll be having our cross-connections moved over today, expect a couple glitches and likely a reset of ADSL connections. All of these interruptions should be brief though.

We'll try to update via Twitter too, but thanks to FIFA, it seems quite unreliable.

Posted by: Myke |

Jun
13
We were down less than Twitter was... ... that's for sure!

Posted by: Myke |

Jun
13
Mostly Done! Bringing up the last of the VPSes now, with the exception of IPMI for colo'd servers, everything should be fully functional again.

Posted by: Myke |

Jun
13
The big step. Going to move the core router, the FreeBSD SAN & VPSes, and the main Power-Distribution Unit... which basically means a lot is happening now.

See you on the other side!

Posted by: Myke |

Jun
12
Moves... Phase 3 is just about done. The fully virtualized servers are moved, one of the power-distribution systems is moved... Jigsaw is running again. Monitoring is coming alive...

Next phase is moving one of the switches between the cabinets, this means a lot of temporary connections in the old cabinet... Here goes nothing...

Posted by: Myke |

Jun
12
Pre-Move Video Tour Myke busted out the iPhone and gives you a quick tour in his first-ever video-blog:

Posted by: Myke |

Jun
12
Phase One Complete - Power is in.
- New switch is installed
- Jigsaw (secondary core router) is moved
- As many rails & cage-nuts are in place as possible
- As much as possible has been moved off of the 20A PDU in the old cab (it'll be the first impactful item to move)
- Lots of disruption coming tomorrow ( Saturday night/Sunday Morning ):

Posted by: Myke |

Jun
08
Can't take the heat...
Chainsaw's Hard Drive temperatures
... So we're being kicked out of the kitchen.

Seems the room in the datacenter we're in is getting too toasty for the HVACs to keep up with, and the space-provider is adding another UPS - we've got to move our gear. Mercifully it's about a 40' move, but it's still a pain and there will be some glitches and downtime. When? "This weekend."

We'll be starting shifting redundant/new equipment over to the new cabinet on Friday, with the big moves happening over the weekend... so VPSes will be down for a bit, then DSLs will be out for a little while, webhosting, etc... etc...

It won't happen all at once, and with any luck, we can keep most of the routing & switching running through out the move. (yay! New switch infrastructure coming online!)

Stay tuned to Twitter, we'll announce which bits & pieces are going when, and keep an eye on this space too, there'll be more posts.

Posted by: Myke |

Powered by NucleusCMS | Ported by VinhBoy | Designed by DemusDesign