ticketing light nercomp sig for lisa

Ticketing LightHow New York University Leveraged Open-Source Software to Manage Security Incidents: 

Ticketing Light How New York University Leveraged Open-Source Software to Manage Security Incidents NERCOMP SIG: Information Security and Trouble Ticketing for Help Desks Brian Smith-Sweeney September 25th, 2006

Outline: 

Outline Getting to RT RT Day-to-Day Formal Incident Response with RT Wrapping up Questions

Slide3: 

GETTING TO RT

NYU Info: 

NYU Info 14 Schools and Colleges 65,000+ users, 50,000 active accounts 50,000 enrolled 16,000 staff 11,000 residential 50,000 hosts on NYU-NET 1GB+ connectivity

History of TSS : 

History of TSS Initially two dedicated staff members Part of a decentralized IT organization Formed by ex-helpdesk staff Focused on reacting to security threats

Old Incident Response: 

Old Incident Response Email-alias-based communication: security@nyu.edu Weekly “on-call” rotation Contacts developed via personal interaction Students: student registry lookup and phone call Admins: “Tracey’s little black book” Ad-hoc response and undocumented process flow

The Problem: 

The Problem Too much manual processing Too much reliance on individual domain knowledge No consistency from incident to incident No long-term tracking and trending of incidents No visibility between staff for a given incident But it worked! Until…

TSS Expands: 

TSS Expands Increased staff Increased incident detection capabilities, including ability to detect mass-compromises Increased need to share and track incident information

ITS has an existing system…: 

ITS has an existing system… NYU ITS Helpdesk used Remedy Enterprise ticketing capability Integrated directly into user database Full-time in-house Remedy developer Workflow already in place

…but do we want it?: 

…but do we want it? Heavy customization = slow updates Developer had to prioritize helpdesk needs Required MS Windows Workflow didn’t fit ours

Well, what do we want?: 

Well, what do we want? Cross-platform, preferably web-based client Drop-in to existing workflow Transparent to users Lightweight, easy to install, easy to setup, easy to maintain Flexibility and easy customization: open-source Security: decent authentication/access control model, encryption in transit Useful API for scripting/automated response

First Try: RTIR: 

First Try: RTIR

How about RT? : 

How about RT? “RT is an enterprise-grade ticketing system which enables a group of people to intelligently and efficiently manage tasks, issues, and requests submitted by a community of users. “ “Every ticketing system sucks. Here at Best Practical, we're really proud of the fact that RT sucks less than everything else out there…”

RT (Request Tracker) Background: 

RT (Request Tracker) Background Created 1996 by Jesse Vincent, leading to the formation of Best Practical Actively developed, actively maintained Commercial support available Widely deployed (10,000+ organizations, including NASA, Merrill Lynch, and Perl) Large, active user and development community

RT: Needs Assessment: 

RT: Needs Assessment

RT: Needs Assessment: 

RT: Needs Assessment

Slide17: 

RT Day-To-Day

Externally-Initiated Events: 

Externally-Initiated Events Regular external notifications Spamcop alerts REN-ISAC notifications AOL spam summary Other external complaints One-off incident notifications Harassment General security inquiry Other

Internally-Initiated Events: 

Internally-Initiated Events Passive anomaly detection IPAudit Darknet Passive signature-based detection (Snort) Active detection (nmap/amap)

Too Much Manual Labor: 

Too Much Manual Labor Detection systems worked great! Responder had to spend days generating notifications TSS needed to improve our signal-to-noise ratio

Enter The Monkey: 

Enter The Monkey The Monkey script: Correlates data from disparate data sources Groups data by host Groups hosts by domain Queries local whois for contact information Takes in standard message text Logs into RT Generates one ticket per domain Frees up staff time for response/remediation

Remedy and RT: A Rocky Relationship: 

Remedy and RT: A Rocky Relationship Both email interactive RT plays nice (keeps Remedy ticket info) Our Remedy only includes its own information, so RT can’t tell tickets are related At best we’re just handing off issues Occasionally we’re duplicating info At worst we’re generating useless tickets

Slide25: 

Formal Incident Response with RT

Formal Incident Response - Communication: 

Formal Incident Response - Communication RT allows us to track two separate communication channels in a single ticket Replies sent to all requestors, usually local system administrator Comments sent only to security responders, or not at all Secured communications in transit (https)

Formal Incident Response - Investigation: 

Formal Incident Response - Investigation Realtime information sharing No handwritten notes necessary Physically disparate investigators can share data Incident managers can monitor/coordinate response

Formal Incident Response - Reporting: 

Formal Incident Response - Reporting Final incident report is easy Copy and paste notes from ticket, or Include and reference the whole ticket log Data retention policy enforcement is easy Different events get different queues Queues can be set to flush automatically

Slide29: 

Wrapping Up

RT isn’t perfect: 

RT isn’t perfect We like that it does not strictly enforce workflow/good practice …you might not! Free text searching Makes reporting difficult at times; is bad for trending systems Is bad for understanding relationships between tickets Breaks some logic Some email processing acts funny (HTML, attachments) Lots of moving parts, and things can break

Our method isn’t perfect : 

Our method isn’t perfect We’re maintaining our own ticket system More work, less efficiency for ITS Yet Another Sensitive Data Repository Still relies somewhat on convention We made RT transparent to the user…this can be bad

The Final Word(s): 

The Final Word(s) Ticketing systems can be really useful for incident response Ticketing doesn’t have to be difficult; your system should match your workflow, not the other way around RT, and in general the open-source software model, works really well for security ticketing and incident response

Questions?: 

Questions?