Telephone Power Dependency : Telephone Power Dependency Do your phones work when the lights go out?
In the old days… : FLGHC May 14, 2009 Ray J. Vaughan, MS 2 In the old days… Your dial tone came from the Telephone company Central Office (CO).
The old telephone ran on voltage that came from huge batteries in the CO.
The batteries were kept charged by either the power company or huge generators with weeks of fuel.
Now : FLGHC May 14, 2009 Ray J. Vaughan, MS 3 Now Today’s technology comes with a price… electricity.
Now we need to provide power to our phones, our phone systems
And to the equipment that makes fiber optic networks possible.
Low Tech Solution : FLGHC May 14, 2009 Ray J. Vaughan, MS 4 Low Tech Solution First the easy solution to the big problem
Educate the Public to get a simple or old phone in the house.
Does your phone have a display? Answering machine? Cordless? It has a second cord that goes to an electrical outlet.
When the lights go out, these phones die.
Excuses : FLGHC May 14, 2009 Ray J. Vaughan, MS 5 Excuses “I have a generator”
Will you be able to call for help if it doesn’t start?
How will you call for the Fire Department if it catches fire?
How many days of fuel will you have available?
Will you run it 24/7?
Excuses : FLGHC May 14, 2009 Ray J. Vaughan, MS 6 Excuses “I have a cell phone”
How will you be charging it?
Will the towers survive?
Message: : FLGHC May 14, 2009 Ray J. Vaughan, MS 7 Message: Have at least one simple phone in the house.
Add this to your agency’s home disaster inventory list.
( and take the analog TV OFF the list… HDTV and EM http://rayvaughan.com/HDTV&EM.htm )
Harder Problems to fix : FLGHC May 14, 2009 Ray J. Vaughan, MS 8 Harder Problems to fix The next problem doesn’t have such an easy fix.
See the Light : FLGHC May 14, 2009 Ray J. Vaughan, MS 9 See the Light Fiber Optic technology allows for huge amounts of voice and data over many miles on a hair-thin piece of glass.
BUT, Glass conducts light, not power.
Something has to convert the digital light signals back into analog electrical signals.
This conversion requires sophisticated electronic equipment that needs power.
Remote Terminals : FLGHC May 14, 2009 Ray J. Vaughan, MS 10 Remote Terminals Covert Fiber Optic signals to regular phone lines
Powered by the local power utility
4 to 8 hours of battery power
Usually no backup power source on site
Sometimes called SLC, Subscriber Loop Carrier
Is this really a problem? : FLGHC May 14, 2009 Ray J. Vaughan, MS 11 Is this really a problem? If there is a long (>4 hours) power outage who will be effected?
Everyone. Calling 911 will be impossible.
You. Your Public Safety Facility may be out of service.
Cellular Sites are interconnected with circuits fed by the same RTs
Wilma : FLGHC May 14, 2009 Ray J. Vaughan, MS 12 Wilma
War Stories : FLGHC May 14, 2009 Ray J. Vaughan, MS 13 War Stories Hurricane Wilma October 2005
South Florida
Weak Category 2 hurricane
$20.6 Billion in damage
36 deaths in Florida
Wide area but minimal flooding
Complete power failure in South Florida
3,241,000 customers, 6,000,000 people without power
Weeks to restore power to many areas
Communication Failure : FLGHC May 14, 2009 Ray J. Vaughan, MS 14 Communication Failure As I had predicted, we had major communication failures.
The area effected by Wilma was so wide, resources available in the area were overwhelmed.
Fire Stations : FLGHC May 14, 2009 Ray J. Vaughan, MS 15 Fire Stations Our generators came on
VoIP phones, delivered by T1, ran until the RT died
Backup Hotline/Fax Line also on RT
Last resort: UHF Public Safety Radio System
Conventional, Simplex available
No common points of failure
K&B Radio Site : FLGHC May 14, 2009 Ray J. Vaughan, MS 16 K&B Radio Site Hub site for South end of the County
T1s to dispatch, other sites failed
Site in In-Cabinet Mode, 1 RX, 1 TX
Long RT failure
Portable generator set up
Fuel shortage
Misconnection
Slide 17: FLGHC May 14, 2009 Ray J. Vaughan, MS 17
Slide 18: FLGHC May 14, 2009 Ray J. Vaughan, MS 18
Outage Time Line : FLGHC May 14, 2009 Ray J. Vaughan, MS 19 Outage Time Line Event
Power outage
4 hour countdown starts
End of wind event, earthquake, disaster
Batteries start to die
No communication
Generators arrive
Fuel shortages
Katrina Facts : FLGHC May 14, 2009 Ray J. Vaughan, MS 20 Katrina Facts Estimates of the damage:
Initially over 1.75 million people without phone service
131 Central office effected and 19 of them either destroyed or heavily damaged, serving a total of 187,000 lines.
Cost to restore/replace: $400-500 million
Data and Photo Source: The Central Office
Slide 21: FLGHC May 14, 2009 Ray J. Vaughan, MS 21
Slide 22: FLGHC May 14, 2009 Ray J. Vaughan, MS 22
Slide 23: FLGHC May 14, 2009 Ray J. Vaughan, MS 23
Verizon Central Office, WTC : FLGHC May 14, 2009 Ray J. Vaughan, MS 24 Verizon Central Office, WTC
Verizon Central Office, WTC : FLGHC May 14, 2009 Ray J. Vaughan, MS 25 Verizon Central Office, WTC
Verizon Central Office, WTC : FLGHC May 14, 2009 Ray J. Vaughan, MS 26 Verizon Central Office, WTC
Verizon Central Office, WTC : FLGHC May 14, 2009 Ray J. Vaughan, MS 27 Verizon Central Office, WTC
Verizon Central Office, WTC : FLGHC May 14, 2009 Ray J. Vaughan, MS 28 Verizon Central Office, WTC
Action Items : FLGHC May 14, 2009 Ray J. Vaughan, MS 29 Action Items Establish a working relationship with your telco technical expert
Identify your critical locations
Sorry, they’re not ALL critical
How will a Telcom outage effect your process?
What’s convenient and what’s mandatory?
Define backup systems
Public Safety Radio
Telecommunications Service Priority (TSP)
Action Items 2 : FLGHC May 14, 2009 Ray J. Vaughan, MS 30 Action Items 2 Start with your most critical. Ask:
Where do these lines/circuits need power to operate?
Your end. You better trust your own power
The CO. Batteries, large generators.
Your concern: the ‘everywhere else’
Is there an automatic generator at each?
Home-Run to CO? Or in a loop?
Take a look : FLGHC May 14, 2009 Ray J. Vaughan, MS 31 Take a look Learn what to look for
Identify the RTs in your area
Look for generators
Ask questions
Your regular Telco Tech is a great resource for how things really are
Types of RTs : FLGHC May 14, 2009 Ray J. Vaughan, MS 32 Types of RTs Above Ground
Underground
Flooding?
In-Building
In someone else’s building
Worst case. You depend on their planning and survival.
Clues that it’s a RT : FLGHC May 14, 2009 Ray J. Vaughan, MS 33 Clues that it’s a RT Power Meter
Hum or fan noise
Thick cables radiating out
Frequent Telco Trucks parking
Address or RT ID stickers
Slide 34: FLGHC May 14, 2009 Ray J. Vaughan, MS 34
Slide 35: FLGHC May 14, 2009 Ray J. Vaughan, MS 35
Slide 36: FLGHC May 14, 2009 Ray J. Vaughan, MS 36
On-Site RT : FLGHC May 14, 2009 Ray J. Vaughan, MS 37 On-Site RT The best solution is to have your RT in your facility
You provide the primary power
You provide the security
Verify fiber route
Route redundancy
Make sure you trust everyone in your loop
Case Study : FLGHC May 14, 2009 Ray J. Vaughan, MS 38 Case Study K&B Radio Site
Now has an on-site RT
Independent of area RT
On our generator
Redundant fiber routes to CO
Slide 39: FLGHC May 14, 2009 Ray J. Vaughan, MS 39
Single Points of Failure : FLGHC May 14, 2009 Ray J. Vaughan, MS 40 Single Points of Failure Better to have multiple:
Remote Terminals
Generators
Fibers and Fiber routes
Copper circuits (if close enough to CO)
Communication Companies
Generators : FLGHC May 14, 2009 Ray J. Vaughan, MS 41 Generators Subject to barometric pressure failure during storms
Over/Underspeed due to wind
Water in air intake
Projectile damage
Check the oil change frequency
Oil & Filters on site
Downtime for changes
Slide 42: FLGHC May 14, 2009 Ray J. Vaughan, MS 42
Flooding : FLGHC May 14, 2009 Ray J. Vaughan, MS 43 Flooding Many RTs and other Telecom facilities in Katrina were flooded.
Fiber can be underwater
Equipment can’t be.
Identify any critical systems in your system that need to be moved higher
Even a CO can be at risk of flooding
Terrorist Targets? : FLGHC May 14, 2009 Ray J. Vaughan, MS 44 Terrorist Targets? RTs could be an easy target for anyone wanting to effect communication in an area.
Most are easy to spot when you know what you’re looking for.
Exposed fiber, cables, power
Even simple vandalism at an RT can put you out of service.
Monitoring : FLGHC May 14, 2009 Ray J. Vaughan, MS 45 Monitoring Your Telco should monitor every RT
Investigate every alarm, even open doors
Roll generator when AC power fails
Not when batteries fail
Regulators: : FLGHC May 14, 2009 Ray J. Vaughan, MS 46 Regulators: Require permanent, automatic generators for all new RTs
Encourage RT clustering
Require response levels
Ratios of portable Generators to RTs
Phase out RTs without automatic power backup
Mandate in-building RTs for all Public Safety buildings
More Information : FLGHC May 14, 2009 Ray J. Vaughan, MS 47 More Information My information site about Powering and Remote Terminals:
PhoneFailure.com