Big trouble with 4thUtility - or how we lost the Internet for a few days

  In August this year 4th Utility took over from VFM as the guardians of the fibre network on the estate.

Last week, from midnight to 04:00 on Tuesday 16th November, 4th Utility undertook some maintenance work.  The time window was planned well in advance, but the maintenance did not go at all as planned; everyone on the estate who uses the 4th Estate fibre service lost their connection to the Internet for a number of days.

What was expected to be a significant but straightforward maintenance task turned into the complete replacement of server room equipment which in turn required the replacement of every router on the estate.  To their credit, 4th Estate quickly realised the magnitude of the issue and asked their Manchester based staff to get on a train to London right away, and to take along replacement server gear and hundreds of new routers.

It turns out that we really do depend on the Internet these days.  Some were unable to watch the film they wanted to see, some were unable to work, some even lost valuable work.  People certainly wanted to get back online as quickly as possible, hundreds of residents queued to get their new router at noon on Wednesday:

At the head of the queue was a desk where 4th Estate staff (Brand Ambassadors!) were handing out new routers and noting which flat got which device.

Over the next day most new routers were connected.  Mostly residents were able to unplug the old and connect the new by themselves, for some a house call from the 4th Estate staff was needed to get things going.  4th Estate staff manned their desk on the highwalk from Wednesday to the following Sunday

The cause of the problem was something like this:  The equipment inherited by 4th Estate has been serving residents for some time and was very much in need of an upgrade.  The maintenance planned for the 16th of November was to keep the current equipment going a little longer while the larger upgrade was rolled out.  The manufacturer of the current equipment felt that the plan, yanking out and replacing some of the equipment, was safe because the service should just fall back to secondary systems ... but what happened instead was the a total failure.

Given the total failure the easiest thing to do would be to install like-for-like replacements of the broken systems, but the nearest were in Dubai!  So, 4th Estate decided to bring forward the larger upgrade and replace the broken kit with brand new server room systems and replace every connected router too.  Quite a task.

Now that the Internet service is restored for most, if not all, 4th Estate customers I'm sure "lessons have been learned" (again).  The biggest for me is to have a fallback plan for our Internet connection (e.g. a mobile hotspot and a data SIM) and of course to make sure all our valuable data is backed up.