bhattacharyya

The Sprint IP Monitoring Project and Traffic Dynamics at a Backbone POP: 

The Sprint IP Monitoring Project and Traffic Dynamics at a Backbone POP Supratik Bhattacharyya Sprint ATL http://www.sprintlabs.com

The IP Group at Sprintlabs: 

The IP Group at Sprintlabs Charter : Investigate IP technologies for robust, efficient, QOS-enabled networks Anticipate and evaluate new services and applications Major Projects : Monitoring Sprint’s IP Backbone Service Platform

Talk Overview: 

Talk Overview The IPMon Project Routing and Traffic Dynamics

IP Backbone : POP-to-POP view: 

IP Backbone : POP-to-POP view POP OC-48 POP : Point of Presence, typically a metropolitan area OC-12 OC-3

Motivation: Need for Monitoring: 

Motivation: Need for Monitoring Current network is over-provisioned, over-engineered, best-effort… Diagnosis: detect and report problems at IP level Management configuration problems, traffic engineering resource provisioning, network dimensioning Value-added service feedback to customers (performance, traffic characteristics) Detect attacks and anomalies

Existing Measurement Efforts: 

Existing Measurement Efforts Passive measurements SNMP-based tools Netflow (Cisco proprietary) OC3MON, OC12MON Active Measurements ping, traceroute, NIMI, MINC, Surveyor Skitter, Keynote, Matrix Integrated Approach AT&T Netscope Network topology and routes Traffic at flow level granularity Delay and loss statistics

Our approach: 

Our approach Passive monitoring Capture header (44 bytes) from every packet full TCP/IP headers, no http information Use GPS time stamping - allows accurate correlating of packets on different links Day long traces Simultaneously monitor multiple links and sites. Collect routing information along with packet traces. Traces archived for future use

Applications: 

Applications Data from a commercial Tier-1 IP backbone Applications of data: traffic modeling traffic engineering provisioning pricing, SLAs hardware design in collaboration with vendors denial-of-service

Measurement Facilities: 

Measurement Facilities IPMON System Collects packet traces by passively tapping onto the fiber using optical splitters supports OC-3 to OC-48 data rates Data Repository Large tape library to archive data Analysis Platform Initially 17 nodes computing cluster SAN under deployment

IPMON Architecture: 

IPMON Architecture Linux PC with multiple PCI buses

Monitoring links at a POP: 

Monitoring links at a POP

Current Status of IPMONs: 

Current Status of IPMONs Currently operational in one major west coast POP on OC3 links Under way in two major east coast POPs for OC3 and OC12 -- (we hope by July 2001) OC48 in preparation for 1 east coast POP and 1 west coast POP -- summer 2001 Future: Sprint Dial-Up Network, more POPs, European network

Practical Constraints : 

Practical Constraints Difficult to monitor operational network : Complex procedure for deploying equipment  POPs evolve too fast  Too costly to be ubiquitous Technology limitations (PCs, disks, etc.) Only off-line analysis is possible Are 44 bytes enough?

Ongoing Projects: 

Ongoing Projects Routing and Traffic Dynamics Delay measurement across a router TCP flow analysis Denial of service Bandwidth provisioning and pricing

Routing and Traffic Dynamics Project: 

Routing and Traffic Dynamics Project Part 1: what are the traffic demands between pairs of POPs? How stable is this demand? Part 2: what are the paths taken by those demands? Are link utilizations levels similar throughout the backbone? Part 3: is there a better way to spread the traffic across paths? At what level of traffic granularity should traffic be split up?

Slide16: 

Motivation Understand traffic demands between POP pairs

POP-to-POP Traffic Matrix: 

POP-to-POP Traffic Matrix Measure traffic over different timescales Divide traffic per destination prefix, protocol, etc. For every ingress POP : Identify total traffic to each egress POP Further analyze this traffic

Applications : 

Applications Intra-domain routing Analyzing routing anomalies Verify BGP Peering Capacity planning and dimensioning POP architecture

Slide19: 

Generating POP-POP traffic matrices

The Mapping Problem: 

The Mapping Problem What is the egress POP for a packet entering the a given ingress POP?

Mapping BGP destinations to POPs: 

Mapping BGP destinations to POPs BGP table Find best Next-Hop Get Unique Next-Hops Map to POP (Dst,Next-Hop) Unique Next-Hops (Next-Hop, Last Sprint Hop) (Next-Hop, POP map) (BGP Dst,POP) Map Dst to POP

Data Processing: 

Data Processing Step 1: Use BGP tables to generate [prefix, egress POP] map Step 2: Run IP lookup software on packet trace using above map Output : single trace file for each egress-POP, e.g. all packets headed to POP k from monitored POP Step 3: Use our traffic analysis tool for statistics evaluation.

Monitored links at a single POP: 

Monitored links at a single POP Core Core Core Peer 2 web hosting ISP Peer 1

Data: 

Data 5 traces collected on Aug 9, 2000

Traffic Fanout: POP level granularity: 

Traffic Fanout: POP level granularity

Fanout: web host links: 

Fanout: web host links

Time-of-Day for POP level granularity: 

Time-of-Day for POP level granularity

Day-Night Variation : Webhost #1: 

Day-Night Variation : Webhost #1 % reduction at night between 20-50% depending upon access link

Summary: 

Summary Wide disparity in “traffic demands” among egress POPs POPs can be roughly categorized as : small, medium, large; and they maintain their rank during the day. Traffic is heterogeneous in space yet stable in time. Traffic varies by (access link, egress POP pair) Hard to characterize time-of-day behaviour 20-50% reduction at night

Routing and Traffic Dynamics Project: 

Routing and Traffic Dynamics Project Part 1: what are the traffic demands between pairs of POPs? How stable is this demand? Part 2: what are the paths taken by those demands? Are link utilizations levels similar throughout the backbone? Part 3: is there a better way to spread the traffic across paths? At what level of traffic granularity should traffic be split up?

IS-IS Routing Practices: 

IS-IS Routing Practices

Is backbone traffic balanced?: 

Is backbone traffic balanced?

What we’ve seen so far: 

What we’ve seen so far Wide disparity in traffic demands between (ingress, egress) POP pairs + Wide disparity in link utilization levels, plus many underutilized routes + Routing Policies concentrate traffic on few paths Question: Can we divert some traffic to the lightly loaded paths?

Routing and Traffic Dynamics Project: 

Routing and Traffic Dynamics Project Part 1: what are the traffic demands between pairs of POPs? How stable is this demand? Part 2: what are the paths taken by those demands? Are link utilizations levels similar throughout the backbone? Part 3: is there a better way to spread the traffic across paths? At what level of traffic granularity should traffic be split up?

Creating traffic aggregates: 

Creating traffic aggregates To address issues of splitting traffic over multiple paths, need to define “streams” within traffic How should packets be aggregated into streams? Coarse granularity: POP-to-POP Very fine granularity: use 5-tuple Initial criterion : destination address prefix

Elephants and Mice among /8 streams: 

Elephants and Mice among /8 streams Stream : all packets in a group with same /8 destination address prefix Traffic grouped by egress POPs Ingress : Webhost Link

Stability of prefix-based aggregates: 

Stability of prefix-based aggregates

Observations about prefix-based streams: 

Observations about prefix-based streams Recursive : /8 elephant has a few /16 elephants and many mice, likewise at /24 level Phenomenon is less pronounced at /24 level Qn : Are elephants stable? Definition: Ri(n) = the rank of flow i at time slot n Di,n,k= | Ri(n) - Ri(n+k) | each time slot corresponds to 30 minutes

Frequency of Rank Changes: 

Frequency of Rank Changes Conclusion : For load balancing, route elephants along different paths

Conclusions: 

Conclusions Monitoring and measurement is key to better network design IPMon : a passive monitoring system for packet-level information We have used our data to build components of traffic matrices for traffic engineering Backbone traffic can be better load-balanced : destination-prefix is a possible (simple) criterion

Ongoing Work: 

Ongoing Work Intra-domain Routing : Choosing ISIS link weights Load balancing in the backbone Flow Characterization Building Traffic Matrices POP modeling