rww bcp msu jun2001

Disaster Recovery Planning at Michigan State University: 

Disaster Recovery Planning at Michigan State University An Overview Presentation by Richard Wiggins, MSU Computer Laboratory With video appearances by: Bruce Alexander, Administrative Information Services Jeanne Drewes, Assistant Director for Access & Preservation, MSU Libraries Tom Atkinson, Department of Chemistry Lt. William Wardwell, Department of Police & Public Safety (DPPS)

Agenda: 

Agenda Welcome and introductions Why Disaster Recovery Planning? Definitions and distinctions Overview of planning process MSU’s Unit Guide to Disaster Recovery Planning Your action plan Questions

MSU's Disaster Recovery Planning Team: 

MSU's Disaster Recovery Planning Team Rochele Cotter, Client Advocacy Office, Chair Diana D'Angelo, CAO Pam Bach, CAO Bruce Alexander, Administrative Information Services Byron Brown, Libraries, Computing & Technology Rich Wiggins, Computer Laboratory Others as assigned

The Name of the Rose : 

The Name of the Rose Example courtesy of Jeanne Drewes Assistant Director, Access & Preservation, MSU Libraries

Quiz: 

Quiz Suppose you work for a major corporation Suppose a disaster takes out your data center All business functions are shut down Payroll, marketing, intranet, Web presence, sales, accounts payable, accounts receivable – all are down Which business function do you want to restore first?

Why Disaster Recovery Planning?: 

Why Disaster Recovery Planning?

Disasters Happen!: 

Disasters Happen! 1994 Cal State Northridge earthquake Greatest disaster to hit a U.S. university Most of campus devastated Stayed open during rebuilding Many classes taught in open air or tents 1997 flood at Colorado State University Entire library under water 1999 fire at Seton Hall Student deaths June 2001: floods hit University of Houston Many buildings flooded Central IT systems knocked out Retirement of modem pool forced Law school severely disrupted Many days to recover enough to teach Many classes now in alternate facilities Research lost Including live animals

University of Houston: Disaster Story Unfolds on Web: 

University of Houston: Disaster Story Unfolds on Web IT systems status -- including destruction of modem pool: http://www.uh.edu/infotech/ 138 classes relocated: http://www.uh.edu/news/flood/room_changes.html General emergency updates: http://www.uh.edu/news/flood/old_updates.html Law School rebuilding committee minutes (VERY illustrative!): http://www.lawlib.uh.edu/news/rebuild.html Houston Chronicle on flood: http://www.chron.com/cs/CDA/story.hts/storm2001/944632 http://www.chron.com/cs/CDA/story.hts/storm2001/944529

Disasters at MSU: 

Disasters at MSU Fires at MSU Chemistry building - 1999 Ag Hall - 1999 Flood at MSU in 1975

Video Sample: 

Video Sample Cal State Northridge Disaster Preparedness Video Prepared for California universities after the Northridge earthquake

Types of Disasters : 

Types of Disasters Acts of nature ("acts of God") Tornado Flood Even earthquake Midwest fault line Accidents Malicious acts Arson Physical destruction Cyber attacks Denial of service, spoofing, cracking

Murphy’s Law and Disaster Scheduling: 

Murphy’s Law and Disaster Scheduling Disasters occur when… Key people are on vacation It’s a holiday You’re in the middle of a major upgrade Other disasters occur They are least convenient

Why Plan?: 

Why Plan? Thinking through scenarios before a disaster… … makes it far easier to recover from a disaster If a disaster does occur: A disaster is declared You open the plan and read it The plan is your cookbook of steps to perform in an orderly fashion

“If Only”: 

“If Only” If only: We’d known where the building water shutoff is We knew that critical root password We had a copy of the client software needed to maintain the server We realized how important that desktop PC on the secretary’s desk really is We had the original source files for… Source code Photoshop PSD files Excel spreadsheets I'd considered the impact of losing 40 years of research

“What If” instead of “If Only”: 

“What If” instead of “If Only” Disaster recovery planning involves asking “What Ifs” to prevent later “If Onlys” What if a critical business function is disrupted due to disaster? How can we recover from the outage? What will recovery cost? What will it take to fully restore services?

Auditors and Disaster Recovery Planning: 

Auditors and Disaster Recovery Planning Recent trend: moving beyond “do the books balance?” … into risk management Auditors (internal and external) are asking: … “Can an institution's vital business functions survive various disasters?” Increasingly, units can expect auditors to ask: Could I please see your disaster recovery plans? Are those plans adequate?

Definitions and Distinctions: 

Definitions and Distinctions

Business Continuity Planning versus Disaster Recovery Planning: 

Business Continuity Planning versus Disaster Recovery Planning Disaster recovery planning: Older term; focus on disaster and recovery Acts of nature Tornado, flood Accidents Fire (and water damage) etc Malicious acts Business Continuity Planning Newer, broader term; more common in recent literature Focus is continuity of vital business functions

Emergency Response and Business Continuity: 

Emergency Response and Business Continuity After a physical disaster, emergency response services are called Police, fire department, ambulances ORCBS (Office of Radiation, Chemical, and Biological Safety) Business Continuity goal must defer to health and safety E.g., personnel should not re-enter a building after fire, flood, chemical spill, etc. …until relevant experts declare it safe

Loss Prevention vs. Disaster Recovery Planning : 

Loss Prevention vs. Disaster Recovery Planning Loss prevention: identify ways to survive events with minimal disruption of operations Disaster recovery planning assumes a disaster, and asks “how do we recover?” During disaster planning, you will no doubt uncover loss prevention opportunities You may wish to implement some of these … or you may make a list for future implementation E.g. use mirrored disks on a critical server

Recovery vs. Restoration: 

Recovery vs. Restoration After a disaster, first you recover, then you restore… Recovery: Vital records and critical systems are recovered Critical business functions are resumed Restoration: Facilities restored Systems restored Resumption of all business functions; “business as usual.” Your plan details both recovery and restoration

Disaster vs. Catastrophe: 

Disaster vs. Catastrophe Goal is to plan for a plausible disaster Plausible likelihood of occurrence Plausible steps to recover Not for the most catastrophic event imaginable Impossible to plan for extreme catastrophes

Business Functions vs. Systems: 

Business Functions vs. Systems Business functions are “what we do” Enroll students; teach classes; pay staff Systems are tools that implement business functions Sometimes a common name is used for both: Payroll is a business function of paying employees their salaries and wages It is also a system of the same name that AIS maintains It is also the name of the office that is responsible for the business function

Y2K and BCP/DRP: 

Y2K and BCP/DRP Y2K was a special case of disaster recovery planning Much of the literature was similar Many institutions found Y2K plans useful for other, non-Y2K disasters MSU units may find their Y2K plans useful in building their disaster recovery plans

Overview of the Disaster Recovery Planning Process: 

Overview of the Disaster Recovery Planning Process

Your Goal Is:: 

Your Goal Is: A plan! A document Printed and bound Typically a 3-ring binder Copies on site and Off site in secure location At homes of key personnel Including: Emergency contact info Location of keys (physical and software) This portion kept confidential

Primary Focus Is IT Systems: 

Primary Focus Is IT Systems Our primary focus with this effort is IT (information technology) systems … that support critical business functions at MSU Other institutional efforts may address other aspects of disaster planning E.g. DPS and tornado response procedures

More Than Just Backup: 

More Than Just Backup Most IT people are familiar with backup strategies Most have implemented same Backup (onsite and off) is essential to being able to recover from many disasters But backup procedures do not constitute a plan per se The plan details how you will use backups to recover and restore

Disaster Recovery Planning as a Project: 

Disaster Recovery Planning as a Project This is a project Like all projects, it will take organization, effort – and time It will require a project coordinator or leader Who will need to tap other personnel as appropriate

Identifying Single Points of Failure: 

Identifying Single Points of Failure Do you have essential resources with no redundancy? Unique hardware Custom software Written by someone no longer on staff Sole-source vendors Key people

People Are As Important as Systems: 

People Are As Important as Systems Who are key people? What skills are needed to support critical systems? Who is uniquely able to execute steps in the disaster recovery plan? Are there others who can back up the key people?

Focus on Facilities: 

Focus on Facilities A disaster may disrupt your ability to use your normal workplace For people and their desktop computers For servers For printing, copying, etc A complete Disaster Recovery Plan anticipates loss of facilities Alternate locations Telecommuting

Partnering: 

Partnering By partnering you can arrange: Offsite backup In event of disaster: Alternate facilities Personnel to loan Hardware to loan Partners could be internal or external Other MSU units E.g., Dept A and Dept B agree to serve as each others’ alternate facility Similar departments elsewhere Facilities contractors

DRP/BCP Literature: 

DRP/BCP Literature Various books, articles, Web sites software packages All emphasize a similar methodology Series of steps define: How to conduct your planning process How to declare a disaster and execute your disaster plan How to test/audit/maintain your plan

Example DRP Book: 

Example DRP Book Disaster Recovery Planning 2d Edition Prentice-Hall, 2000 By Jon William Toigo

Basic Steps in Disaster Recovery Planning: 

Basic Steps in Disaster Recovery Planning Inventory/identify unit’s business functions and systems Assess risks Identify critical functions and systems Develop plan Who will do what using what resources to recover Prepare disaster recovery notebook Response (in event of disaster) Declare disaster Execute steps in plan Recover & restore Test / audit plan Maintain plan

Business Impact Analysis: 

Business Impact Analysis Business Functions Student Payroll Academic Advising Federal Agency Reports Classroom Teaching Research Projects

Business Impact Analysis: 

Business Impact Analysis Business Functions Systems & Other Resources Student Payroll Paper forms PC / spreadsheet in dept office Custom software on dept server Process to upload records to Payroll

Business Impact Analysis: 

Business Impact Analysis Systems & Other Resources Risks Fire or flood in main office Network outage Software failure; programmer gone Payroll clerk quits suddenly Business Functions Student Payroll

Business Impact Analysis: 

Business Impact Analysis Impact Paper forms PC / spreadsheet in dept office Custom software on dept server Upload process to Payroll Risks Systems & Other Resources Business Functions 1 day = minimal 1 week = serious 1 month = extremely serious

The Time Factor: 

The Time Factor You will be analyzing and evaluating: What your critical business functions are What systems support those functions What disasters might interrupt your critical functions and systems You also must consider the time factor What is the impact of a potential outage… … based on duration of that outage?

Weighing Criticality and Duration of Disruption: 

Weighing Criticality and Duration of Disruption For example: If you can't admit students for one day, minimal impact If you can't meet grant obligations for six months, major impact

MSU’sUnit Guide to Disaster Recovery Planning : 

MSU’s Unit Guide to Disaster Recovery Planning

Phase I – Information Gathering: 

Phase I – Information Gathering

Step 1. Organize the Project: 

Step 1. Organize the Project Who will lead/coordinate the project? Who else needs to be involved? When does it need to be done? What tasks are involved in creating your plan? What are the objectives and scope of the plan?

Step 2. Conduct Business Impact Analysis: 

Step 2. Conduct Business Impact Analysis What are the most critical functions and systems in your unit? What would be the impact if they were severely interrupted? What is the impact if there is a disruption? $$$ cost? Students not enrolled? Classes not taught? Research not performed? Grant obligations not met? Threat to human life or safety?

Step 3. Conduct Risk Assessment: 

Step 3. Conduct Risk Assessment For each critical system or business function: Where is the critical system or function performed? What are the site risks? What is the probability of this area being severely interrupted?

Step 4. Develop Recovery Strategy: 

Step 4. Develop Recovery Strategy How will you operate during a severe disruption to insure all critical functions can be performed? How will you get your unit back up and running? Prioritize: Which system(s) will you restore first in case of disaster? Note: This step is strategic – a bird’s eye view. Step 7 (below) is the detailed view.

Step 5. Review Onsite and Offsite Backup and Recovery Procedures: 

Step 5. Review Onsite and Offsite Backup and Recovery Procedures Are you backing up critical information and systems? Where are the backups located? Will your backups survive an incident in your building? E.g. fire, water damage, etc.

Step 6. Select Alternate Facility: 

Step 6. Select Alternate Facility Do you have a location to perform work in the event your facility is destroyed or rendered unusable? Note: Partnering can be very useful here.

Phase II – Writing and Testing the Plan: 

Phase II – Writing and Testing the Plan

Step 7. Develop Recovery Plan: 

Step 7. Develop Recovery Plan Produce a document (notebook) describing in detail: All business functions All systems supporting those functions All steps required to: Recognize a disaster Mobilize key people (emergency contacts) Initiate recovery Keep operations going during recovery Fully restore services

Step 8. Test the Plan: 

Step 8. Test the Plan Does the plan work? Are all contact numbers listed and critical areas detailed?

Phase III – Maintaining and Auditing the Plan (Ongoing): 

Phase III – Maintaining and Auditing the Plan (Ongoing)

Step 9. Maintain the Plan: 

Step 9. Maintain the Plan When changes occur in the work force, system, equipment, or process the plan needs to be updated to reflect these changes.

Step 10. Perform Periodic Audit: 

Step 10. Perform Periodic Audit Have someone outside of your unit review and assess your plan. Note: if the author of the plan tries to audit it, gaps will go unnoticed. Partners can be very useful here.

Ten Steps: That’s It!: 

Ten Steps: That’s It! If you follow these steps, the end result will be a viable plan that is maintained over time. The Step-by-Step Guide includes worksheets to assist in each phase of your planning

MSU DRP Web Site: 

MSU DRP Web Site Under construction http://DisasterRecoveryPlanning.msu.edu Pointers to Web resources Information on training Materials The Unit Guide Overview Step-by-step instructions Worksheets to help units do planning

Your Action Plan: 

Your Action Plan Download and review Unit Guide Consult with management and colleagues as to planning team(s) Follow the Step-by-Step Guide Inventory Assess Write plan Test / Audit Maintain Distribute your DRP plan as appropriate

Interview: 

Interview Jeanne Drewes Assistant Director for Access & Preservation MSU Libraries

Interview: 

Interview Bruce Alexander Associate Director Administrative Information Services

Presentation: 

Presentation Tom Atkinson Senior Academic Specialist Department of Chemistry

Presentation: 

Presentation Lt. William Wardwell Emergency Management Coordinator Department of Police & Public Safety

Questions: 

Questions