Lessons Learned from Derecho 911 Problems

by ARLnow.com August 20, 2012 at 11:30 am 3,520 16 Comments

Arlington County is using the 911 problems following the June 29 derecho as a learning opportunity.

In the days following the storm, which left 1 million customers without power in Virginia, Arlington and Fairfax counties experienced numerous problems with its 911 service, which made 911 unreachable for many emergency callers; some callers got a busy signal after calling 911, others heard nothing.

The problems were traced back to Verizon’s local communications backbone. In a report released last week, Verizon said the 911 problems started as a result of power outages.

Verizon’s central phone facility in Arlington lost power after the storm. It operated for a few hours on battery power, but a generator at the location failed to start (due to fuel line problems) and the facility lost power at 5:00 a.m. on June 30, after the batteries drained. Although Dominion restored power to the facility at 12:45 p.m., it took time for Verizon to recover its “telemetry” systems, which allow it to see and diagnose problems in its phone network, which had been been damaged by falling trees pulling down phone lines.

In Arlington, 911 service was spotty for days, but was deemed restored and stable by July 4.

In its report, Verizon said it has learned lessons from its 911 failure and will be improving its generator maintenance and redesigning some of its network systems to improve redundancy and reliability. Arlington County says it, too, has learned lessons from the experience.

Jack Brown, Arlington’s director of emergency management, said he’s hopeful that a 911 failure will never happen again. Should it happen, however, he said the county will have a more defined playbook of how to handle the situation.

During the days after the storm, Arlington advised those with emergencies to call the county’s non-emergency line at 703-558-2222. If all else failed, the county also staffed its fire stations so that residents could walk in and report an emergency. A “couple of people” did end up resorting to walking to fire stations, according to Brown.

Through the county’s efforts and somef luck, Brown said no one in Arlington was seriously harmed as a result of the 911 failure.

“We are very lucky that we didn’t have any life-threatening emergencies that couldn’t get through during that time,” said Brown. “We were very vulnerable during that period.”

In Virginia, a state panel is currently investigating the 911 failures, Brown said. Northern Virginia congressmen also called for the FCC to investigate the problems.

Flickr pool photo by ddimick

  • G Clifford Prout (now moderated for extra purity)

    Don’t forget to keep paying your 911 fee on your phone bill.

  • Mick Way

    Blocked fuel line????

    Don’t these guys test-start the generators on a regular basis (like weekly)???????

    Heck, the bank in my building does (I sit 2 floors up from the damn things). This is unbelievable.

    • Dude Where’s My Car

      +1 you don’t just leave diesel generators for something like 911 sitting around for months and hope they start up when you want them to. If the Giant supermarket can have this figured out and run everything except perishables off a generator after the derecho, so should Verizon.

      Data centers often have multiple (diverse) power feeds coming from different electrical substations. The different power busses are independent and have their own generators, batteries etc. So it’s not just a matter of “oh a fuel filter was clogged in this ONE generator.” You’d think that the same level of engineering that goes into making sure Facebook stays up should go into E911.

      Then again, just adding diesel generators isn’t necessarily the answer, as Fukushima showed. If you have multiple diesel generators, it doesn’t help if they’re all under water.

      But the generator problem was fixed within hours. The “telemetry” network problem lasted FOR DAYS and there is no information on what it is, what caused it, or how it could be mitigated in the future. On the face of it, it seems absurd to say they couldn’t fix problems in the network because the part that identifies problems in the network wasn’t working. 😛 There’s these things called “human beings” that can report network problems too…

      We dodged a bullet here, thank God, but let’s REALLY identify the “lessons learned” and put them out for public consumption. Some handwaving about improving the generator maintenance isn’t going to cut it.

    • Suburban Not Urban

      That’s the diff between the public and the private sector – the private sector says – oh if this happens we go out of buisness and I loose my job so they make a practical – pretty fool proof system; the public sector says says oh mea culpa; we’ll learn leasons and goes on cashing their pay check.

      Also I agree with below – the generator didn’t work – get it back – OK – back on line in a couple of hours – but 5 days to fix what ever when down back on line – that’s just fragile/un-reliable for an emergency response system – sounds like someone taking a check/contract – but spending as little as possible to provide the service and pad their margin.

      • JJ

        Last time I checked Verizon is private sector.

  • dang, being a 911 operator would be a pretty wild job. hadn’t really thought about that since i was like 6 yrs old until now.

  • JimPB

    Would natural gas powered back-up generators avoid the problems of clogged fuel lines and the inability to obtain gasoline or diesel fuel from underground tanks for refueling when there is a widespread and persistent electric power outage?

    • South Side Chris

      It depends on the nature of the emergency and the availability of natural gas in your area. Plus the infrastructure installation costs are considerable for something that doesn’t get used very often. Diesel is cheap and cheerful plus easy to maintain and gives you a good amount of electricity for your investment. You need to maintain it but that doesn’t seem to have happened here. Plus diesels require less fuel care than a gasoline generator. Gasoline generators tend to give less electricity but the fuel is easier to come by. Even with stabilizers, gasoline breaks down comparatively quickly. Gasoline is great for small generators that run a few appliances at home.

      Companies go for Diesel most of the time because it’s tried and tested technology that is fairly easy to maintain, fix and operate. It gives good steady output, is self contained and you can easily find people with the skills to work on it.

      What does impress me about the Verizon situation is how long their batteries lasted. That amount of run-time represents a considerable investment in a solution that has to be completely replaced every few years.

    • brown before green

      Yes, Jim, natural gas generators are a viable option for those who have ready access to natural gas. But even they aren’t immune from interruption, as sometimes natural gas quality suffers in severe cold weather (as happened around here about 15 years ago). And they still need to be tested (exercised) routinely.

  • YTK

    Yeah sure–“A lesson, A lesson” — they ALWAYS say that– EVERYONE says that and then they forget. Or the ones who replace them in the future have to reinvent the wheel all over again..

    • WLO

      Frankly, I do not see the exposition of any lessons learned in this article.

  • Not quite, Jack Brown

    “Through the county’s efforts and some luck, Brown said no one in Arlington was seriously harmed as a result of the 911 failure.”

    How about Dylan Shawn Cooper, who lives just inside Fairfax County but is served by Arlington Fire Department’s Station 6? He was struck by a downed power line and people trying to help him couldn’t get through to 911. He died of his injuries two weeks after the storm.


  • soarlslacker

    It would be really prudent for Verizon and Arl Cty to get their upgrades and fixes in place and then TEST THIS STUFF. Any private company would test it. Any carrier whose head was not up its derriere, would test it. They can easily plot a small window when they have the lowest volume of traffic, announce the test and then do a level one test. Does it work? If they pass that, they can do an extended test which would mimic the conditions in a real, extended emergency. This is not rocket science. MCI did this type of testing for business customers frequently and for it’s own nextwork. Maybe this MCI philosophy did not get acquired by Verizon.

  • Not quite, Not quite

    The young man that died was in Fairfax County; in an area not served by Arlington #6 as the closest firehouse. That area is run first by Fairfax Fire and not Arlington. Arlington sent units, but only due to the closest Fairfax units being unavailable. The call load for all firehouses throughout the region was tremendous and units were responding well outside their regular area. Regardless, this 911 call was not in Arlington and not in an Arlington firehouse first due area.


Subscribe to our mailing list