Deeper Implications of the Great PG&E Outage
It was 11:10pm on the night of October 9, 2019. Here along the Monterey Bay the weather was calm, cool, and a bit damp. The lights were on, the internet was working, the car was charging; everything was normal.
At 11:11pm things suddenly changed.
The lights went out.
Internet service dropped.
Cellular signals vanished.
Telephone dial tone went silent.
Had an emergency happened we would not have been able to call 911.
It was a most inopportune time for 911 emergency service to fail.
This power outage lasted only two days.
It could have been much worse. PG&E estimated five days or more for the outage, with additional days for recovery.
911 service could have been down for more than a week.
PG&E plans future "Public Safety Power Shutoffs". Let's consider this event as a harbinger of things to come.
This note digs more deeply into that failure and the likelihood that our notion of emergency service has badly eroded.
The Notion Of 911 Emergency Service
Time is critical during an emergency. Life and property may depend how fast emergency services can be summoned.
Most countries have established emergency services via a special telephone number. In the US this is "911"; in other countries it may be "112" or "999" or something else.
It has been broadly recognized that this life preserving service must be as robust as possible, that it not disappear during foreseeable emergencies.
Recognizing its value, the US made significant financial, technical and legal investments into the 911 system.
For example 911 service contains a location element that can provide detailed location information of an incoming call to the emergency services center.
Sometimes even mobile phones that are not otherwise active can dial 911 services.
Decades of work have gone into the creation of the emergency services system. But "Public Safety Power Shutdowns" have undermined that work. The new reality is that the 911 emergency service system can vanish in a millisecond.
Electrical Power
911 services do not work without electrical power.
Power is needed for the entire chain of equipment that attaches the user to the call center.
Loss of any link, in that chain will disconnect the person in need from the emergency service center.
Ma Bell And The Central Office Powered Telephone System
A few decades ago, when wired telephones were the norm, the telephone company provided all the necessary power. That power was delivered over the same copper wires that attached the customer's telephone. Telephone central offices were equipped with significant amounts of backup power so that they would weather long power outages and, at the same time, maintain working telephone service.
Things have changed.
Modern Telephones Systems
Today many people are "cutting the cord" by abandoning the old Ma Bell style of hard-wired telephones with central-office supplied power.
Today's communication devices include telephones, computers, mobile phones, tablets, Amazon Alexa, and the growing number of home automation and Internet of Things widgets.
None of these use central office supplied power.
All use local power, the kind of power that goes off during a public safety power shutoff.
And most of these are dependent upon a number of intermediary devices that themselves depend on a source of electrical power. If any one of those intermediaries goes down then the customer is cut off, even from 911 service.
What this means is that modern telephone systems and social media services like Twitter, Instagram, and even basic text messages, are dependent on a source of power to the user devices themselves and to every intermediary between the user and the service.
Many devices have small batteries. But those will be depleted after a few hours.
In addition, many devices have no batteries at all, or have batteries that have aged and are no longer useful.
In this new era of "public safety power shutoffs" utility companies advise that electrical power may be off for as long as five days, plus additional days for recovery.
Even devices with good, fully charged batteries will not last for that duration. They will stop working long before the power comes back on.
Provider Monitoring
Cable systems are based on a standard called DOCSIS. This standard give the cable provider a lot of tools to monitor and manage the customer premise equipment.
What is not known is whether providers use the capabilities provided by DOCSIS to check whether backup batteries are good or in need of replacement.
What About Cellular?
Cellular (mobile) devices depend on at least two elements having power: the mobile phone device itself and a cell tower.
The phones have batteries, often sufficient for a day or so if not continuously used. And cell towers sometimes have backup batteries or diesel powered generators with fuel for perhaps a few days.
However, usually there are more elements involved.
In areas with hills or other obstacles mobile providers often install "femto cells" into offices of homes or people who complain of bad service. These are essentially small cell phone towers, with limited range, and that use the office or home internet service as a way to carry the voice data back to the mobile provider.
These femto cell devices will go down when the user's power goes down, thus leaving the user dependent on exactly those same poor (or non-existent) signal conditions that drove the installation of the femto cell in the first place.
And for full sized cell towers there is often a sequence of network devices — switches, routers, amplifiers, etc — that all must have power to be up and working. This is often called the "backhaul".
It Could Be Worse With 5G
Present day tech media is filled with hyperbolic promises about 5G.
What is not mentioned is that one of the ways that 5G works is through smaller, meaning more, radio cells. That means more things that will need power; that means more things that will go down when the power fades.
Many of these 5G devices will not reside in centrally located carrier centers, rather they will be scattered around on everything from street lamp posts to wiring closets in office buildings. Getting backup power to all of those locations will be both hard and expensive. And often it may be entirely overlooked.
It is not unreasonable to anticipate that 5G mobile communications may be more vulnerable to power outages than 4G or earlier systems.
The Failure Sequence
What can we anticipate over a long duration power outage, such as the five day periods mentioned in PG&E's warning?
Hard wired, plain old telephone service (POTS) phones hooked to old fashioned central office plants may continue to work through the darkness.
Internet based phone systems, or systems based on cable TV or fiber-to-the-home technologies, will probably fail in one of two ways:
Some phone (and 911) services will fail instantly, because somewhere in the chain of necessary devices, backup power will be lacking or backup batteries will have died and nobody noticed
Other services will begin to slowly crumble as the hidden elements, such as routers and switches, begin to go offline because their backup batteries have drained or the fuel to their generators has run out.
Given that in the past most power outages have been short, usually less than a day, we can expect that many providers have not provisioned more than a day's worth of backup power.
Consequently, we can envision that customer phones (including 911 service) that survive the immediate impact of the power shutdown, will begin to fail after several hours. Followed by a near total shutdown within 48 hours.
What Can Be Done
There are two obvious remedies, but both are flawed:
Fallback to hard-wired, central office powered, systems.
More backup power.
The first of these remedies is unlikely. Utilities and providers are not merely moving away from hard copper loops under our streets, they are often removing or disabling them for cost, maintenance, or marketing reasons.
The second remedy is possible. However, battery backup is both expensive and can be physically large (and heavy). In addition, backup systems do require periodic maintenance.
Some of this can be alleviated as more people have local backups for their homes and offices. These are often called "microgrids" and are exemplified by products such as the Tesla Powerwall.
However, microgrids do not address power failures in the network/phone providers' equipment, such as cable head-ends or cellular towers.
One can moderate, if not cure the problems, by deploying equipment that has adequate built-in backups and means to warn users when those batteries are aging and in need of replacement.
This is exactly what is being required by the US Federal government. However, some providers seem to be ignorant of those requirements or are evading them.
There is another possible solution, but one certain to draw loud expressions of pain from the providers, which is to impose legal liability for physical injuries that could have been ameliorated had 911 services not become unusable.
FCC Requirements
The United States Federal Communications Commission (FCC) has published a 24 Hour Home Backup Power Requirement
This rule does not obligate providers to put 24 hour backup power into their customer devices. Rather it requires providers make such backup available, for a price.
Despite the rule, customer premise devices, such as cable modems, provide less than eight or hours of backup battery, if any at all.
Non-Scientific Survey Of Provider Adherence To FCC Mandated Requirements
In the few days since the PG&E outage we performed an entirely non-scientific, statistically invalid, undersized sample, survey to see if providers are meeting the FCC's requirement to offer 24 hour backup.
We did not find any instance in which the provider was meeting the FCC requirement.
Even If Those FCC Requirements Were Met They Would Be Inadequate
The FCC requirement is a good first step (assuming that our survey is inaccurate and that, instead, providers are, in fact, honoring the requirement.)
However, as an extra-cost option, not every consumer would buy-in. In fact, given the highly price competitive market for communications, it is likely that few would buy in.
One solution to guarantee a workable backup requirement, would require mandating backup power in all devices. Providers will object. However, at one time in the automobile industry, anti-lock brakes and seat belts in became mandatory. Sometimes we have to make some choices. Remember, the person who needs 911 emergency service may not be the person paying the bills.
As an alternative, the power utilities could be required to pay a form of reparations for every hour they have power turned off. That money could be used to offset the cost of backup power.
There is an additional problem. Providers may have not provisioned sufficient power backups to back-haul. This new era of week-long blackouts was not readily foreseeable. Many providers have built-out their systems with coverage only for the shorter outages that have been the norm before now.
Providers will have to bear increased expenses to build and maintain these expanded services. This will end up increasing customer monthly bills.
Excuses
Communications providers are skilled practitioners in the art of evasion by definition.
We can anticipate that many providers of gear that is used as part of 911 services will try to define away their role or responsibility.
For example we can foresee that some providers will try to say that they are not providing "Plain Old Telephone Service" (POTS) but are instead providing some sort of "Voice over IP" (VoIP) service, with the implication that only POTS is subject to a power and continuity requirement.
We can also expect providers to convince us that provision of power sufficient to cover a five day public safety power outage will be too expensive and will force a large increase in prices to consumers.
Or providers might simply ignore FCC or State public commission rules.
Conclusion
We are now in an era in which continuous electrical service is no longer a given.
Rather, electrical utilities, such as PG&E or Southern California Edison, will avoid fire risks by calling a public safety power shutoff and turning off our electrical service for as long as a week.
There has been little to no concern how these outages will cascade and affect other utilities, particularly the 911 emergency service.
The safety of the public is at risk.