Server Security Patch Management at Microsoft: Server Security Patch Management at Microsoft Sharing the Microsoft IT experiences Published: April 2004


Solution Overview: Situation Security vulnerabilities can lead to loss of revenue and intellectual property Solution SMS 2003 is key tool in Microsoft IT patch management process Benefits Automated deployment of security updates and applications Central reporting and administration More accurate and efficient patch management Reduction in manual effort to patch servers Solution Overview


Products and Technologies: Products and Technologies Windows Server 2003 SQL Server 2000 SP3a SMS 2003 MBSA 1.2 Change management database (IT Configuration database)


Challenges: Large, highly dynamic environment Security 2,500 attacks, probes, and scans daily Over 125,000 virus-infected messages quarantined monthly Unique IT environments for product development, testing, support, and research require special security Technology-literate staff 95% with local administrator right to their desktop Challenges


Business Benefits: Automated security update and application deployment Enforcement within prescribed timeframes Minimized unplanned downtime Central reporting and administration Clear communication path More accurate and efficient patch management More updates, fewer administrators, less time Reduction in manual effort to deploy updates Automated tools, fewer scripts Business Benefits


Background: Background Three main data centers with centralized IT operations Tiers of support for fully managed servers Help Desk Data Center Operations Infrastructure Support and Advanced Diagnostics and Debug teams Engineering


Server Patch Management Architecture: 1 Central Site Server Windows Server 2003 SQL Server 2000 SP3a 10 Primary Site Servers Windows Server 2003 6,000 Windows Server 2003–Based Servers Running SMS 2003 Advanced Client with Advanced Security Server Patch Management Architecture


Server Patch Management Process: Team Roles: Server Patch Management Process: Team Roles MSRC Releases security bulletins Corporate Security Assigns deployment priority Data Center Operations Manages data centers Hosts SMS infrastructures Patches servers


Server Patch Management: Phases: Phase 1: Monitoring for security bulletins and updates from Microsoft Process of deploying update to servers begins after update is released Two schedules, one deployment/enforcement process Server Patch Management: Phases


Server Patch Management Process: Phases: Phase 2: Determining the risk level MSRC - Critical, Important, or Moderate Deployment scheduled based on adjusted MSRC rating Security Update Inventory Tool helps determine which servers are vulnerable MBSA scans for missing/installed updates Server Patch Management Process: Phases


Server Patch Management Process: Phases: Phase 3: Testing Deploying synthetic patch to test deployment success Monitor success, investigate and fix failures Server Patch Management Process: Phases


Server Patch Management Process: Phases: Server Patch Management Process: Phases Thursday Friday Saturday Sunday Thursday Hour 1 Hour 2 Friday Hour 3 Saturday Hour 4 Sunday 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 12 A.M.– 4 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 4 A.M.– 8 A.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 8 A.M.– 1 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 1 P.M.– 4 P.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 8 P.M.– 12 A.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 4 P.M.– 8 P.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. 8 P.M.– 12 A.M. Standard Deployment Emergency Deployment Phases 4–7: Deploying the patch


Server Patch Management Process: Phases: Phase 8: Reporting Determine success of deployment and degree of voluntary patching Advertisement Status Viewer Identify compliance levels Compliance by Software ID report Server Patch Management Process: Phases


Lessons Learned and Best Practices: Lessons Learned and Best Practices Establish a change advisory board To control planned downtime, use a change control database Target update distribution according to pre-determined maintenance periods Streamline the SMS 2003 installation by enabling only necessary features Aggressively monitor and manage SMS client


Lessons Learned and Best Practices: Lessons Learned and Best Practices Suspend monitoring during patching Make status self-serve through a website Communicate the rollout schedule to the organization Assign routine software distribution points


Lessons Learned and Best Practices: Lessons Learned and Best Practices Monitor bandwidth when sending updates between SMS sites Use the Update Wizard Advertise the update to client computers


Lessons Learned and Best Practices: Lessons Learned and Best Practices Test the impact of the update Create a reference collection Test basic functionality, then add levels of complexity Model the test lab on the production environment Deploy updates in timed phases Consider exemptions from forced updates Set forced updates to coincide with off-peak hours


Lessons Learned and Best Practices: Lessons Learned and Best Practices Identify computers that were off the network Baseline the environment Begin with accurate inventory Keep baseline simple and enforce rigidly Bring sub-baseline computers into compliance Carefully consider servers that exceed baseline


Lessons Learned and Best Practices: Lessons Learned and Best Practices Establish enforcement policy Plan disaster recovery Implement SMS 2003 Advanced Client throughout the enterprise Consolidate updates into service packs Continually improve the process


Lessons Learned and Best Practices: Gather Performance Statistics (Example) Patching activity Ratio of rejected patch RFCs Ratio of emergency patches Patch success ratio (per patch) Number of support incidents (per patch) Cost of downtime, productivity loss, or lost business transactions per update Time from test success to 60% saturation deployment Identify time from 60% to 80% saturation deployment Identify time from 80% to 90% saturation deployment five per month one out of six one out of four 97% Nine $25,000 1: 75 hours 2: 12 days 3: 30 days N/A 1: 25 hours 2: 10 days 3: 30 days Example Trend Measurement Actions Baseline for comparison Document RFC completion requirements; educate staff on RFC completion requirements; enforce RFC completion through Change Log tool 1: 10 hours 2: 10 days 3: 30 days Circumvent network bandwidth and bottleneck issues; resolve policy and compliance issues; resolve notification failures or miscommunications; note maintenance period changes for renegotiation; note workload and cycles for capacity planning purposes. ↓ ↓ Implement mitigation strategies and tactics to reduce attack surface. ↑ Systematically document and incorporate failure modes into testing scheme. ↓ Produce reusable workarounds; bring rogue systems into baseline compliance (upgrade, service pack, etc); provide self-help on website; push self-help to users in e-mail, voice mail, or other notification mechanism; better prepare and educate helpdesk. ↓ Process improvements that lower this cost improves profitability; use this number to guide patching timelines. Circumvent network bandwidth and bottleneck issues; resolve policy and compliance issues; resolve notification failures or miscommunications; note maintenance period changes for renegotiation; note workload and cycles for capacity planning purposes. Circumvent network bandwidth and bottleneck issues; resolve policy and compliance issues; resolve notification failures or miscommunications; note maintenance period changes for renegotiation; note workload and cycles for capacity planning purposes. ↓ ↓ ↓ Lessons Learned and Best Practices


For More Information: For More Information For additional information about how to deploy, operate, maintain, and support SMS, visit http://www.microsoft.com/smserver/ For details about MSM and MOF, visit http://www.microsoft.com/technet/itsolutions/


For More Information: For More Information Additional content on Microsoft IT deployments and best practices can be found on http://www.microsoft.com Microsoft TechNet http://www.microsoft.com/technet/itshowcase Microsoft Case Study Resources http://www.microsoft.com/resources/casestudies


Slide23: This document is provided for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. © 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Microsoft Press, Visual Studio, Visual SourceSafe, Windows and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.