noc2

NOC Theory and Practice: 

NOC Theory and Practice Part II: The Nuts and Bolts (The Practice Of Making a Good NOC)

DISCLAIMER: 

DISCLAIMER I am not affiliated with any corporation. My views are my own.

Intro - What’s In A NOC?: 

Intro - What’s In A NOC? Purpose - What Type of Company Are You? People will always be your key asset. Systems make the job of people easier. Managers assist in identifying gaps in both. Processes allow for flexibility in operations while still maintaining standards. And, last but not least, the facility is where you do your work.

The Facility - Think Data Center: 

The Facility - Think Data Center Location, location, location. Security Environmental Concerns This applies to both your inside and outside environment. Ergonomics Reliability Sources of Labor

The Goals of a NOC: 

The Goals of a NOC Our conception is: At the bare minimum, you will receive requests from your customers You will process those requests in some fashion (engineering, handoff, ...) You will communicate resolution to the customer, and/or keep customer updated on status

Goals II: 

Goals II NOC as first, second, or third tier support. Conception of NOC as “do everything” group The engineers should be smart at designing, configuring, etc. - NOC should be divided into categories of engineers able to resolve issues A smart issue routing process makes this a very effective model

The Process: 

The Process Receive events from your “inputs” (phone calls, network monitoring, etc) Process inputs into a problem to be resolved Route the problem to the correct individual Resolve the problem as appropriate Communicate status continually. Record the problem for later review and training.

The Key to Success Is: 

The Key to Success Is By constant re-examination of your operation, your employees, and what inputs you are getting, you can reach incredible efficiency. Communicate this to other teams and your operations will prosper.

The Facility (cont.): 

The Facility (cont.) Security applies to keeping other people out, but it also applies to making a safe arrival for your employees, at any hour. Environmental concerns: Just like a datacenter, consider climate control and earthquake/act of god resistance. Disaster planning rules should be in effect. Sources of labor: Not just engineers, but builders.

A Word on Portability: 

A Word on Portability As Matt suggests, portability is critical. Obvious resources: Computers, communications, phones. Not obvious resources: Vehicles and quick transport/assembly, good cases for shipping or relocation, breakdown/reassembly schedules. Multiple everything. Make sure your resources are distributed. If your network is large, your footprint of mobility should be large.

Hiring the Right People: 

Hiring the Right People Skillset but more importantly, flexibility Stress Scheduling Stability NOCs are hard things to staff appropriately. And good people are hard to retain.

Hiring (cont.): 

Hiring (cont.) Make sure they have lives. But know what you can’t ask. Can they talk? How are their soft skills? Do they get flustered easily? And remember, org charts suck. Classify people according to their skills, not according to some hierarchy which has no meaning outside of some middle-management mind.

Keeping Your People Happy: 

Keeping Your People Happy Environment Ergonomics Creature Comforts We’re not kidding. Give some thought to a 24x7 outlook. This means beds, showers, food, and climate.

Environment - People: 

Environment - People Climate controlled centers get cold at night. Make sure you have 24H control over your space. When thinking about climate, keep in mind your working conditions are slightly different in a NOC. 2:00AM to 4:00AM are the critical hours.

Ergonomics - People: 

Ergonomics - People Hire a good ergonomic consultant. Positions of lights, phones, keyboards, monitors, pens - all this matters. Observation is key. Your engineers will show you what the problems are. (Cameras) Seating. If you’re in a chair for eight hours at a stretch, don’t you want it to be a comfortable one? Screen space.

About People. About NOCs.: 

About People. About NOCs. It’s a messy business. It’s high stress. It’s easy to let things get out of hand. Good management qualities in a NOC: cool heads methodical approach to solving problems having been on the customer side micromanagement sucks.

Let’s go over retention.: 

Let’s go over retention. Retention of NOC staff is 100% harder. Very few people live for this kind of job. So, innovate. Think outside the box. People want to grow. Know what your people want to do when they grow up. Remember your core focus. Do operations, not sales, or account services, or anything else. At the same time, make sure your folks are informed on everything. Everything comes out in the wash. All the dirty clothes come to NOC.

Interviewing NOC folks: 

Interviewing NOC folks You are interested in three things. The candidate’s willingness to learn. The candidate’s 1-2 year outlook on their career. The candidate’s ability to work funny schedules. They are interested in three things. Will you offer them growth potential? Can you give them enough perks to enjoy their work? Do you have your stuff together? Are the procedures well-documented? Will they be called in outside their shift? Will they ever wonder who to go to?

Productivity: 

Productivity Momentum is key. Keeping your people busy is the most important thing towards maintaining productivity. The other component to momentum is keeping them working on different things. This makes them better engineers, and helps feed ego. “Everything at your fingertips” - your engineers should never have to make a customer wait more than four minutes on hold for anything. If it has to wait longer than that, tell the customer why, and mark it as something to fix.

The View of the Customer: 

The View of the Customer A customer wants Professionalism: your engineers must treat the customer as if they are paying you a lot of money and the engineer wants the customer to continue to do that. ETR: Your customer wants to know when the problem will be fixed. “I don’t know” is not an acceptable answer. The ETR is the first thing you give to a customer and the one thing you update him on at least every hour (or less if they prefer).

The View of the Customer II: 

The View of the Customer II A description: Concise or not, the customer wants to know what the problem appears to be. Remember your focus. It is not (typically) your responsibility to engineer your customer’s network for maximum reliability. Your responsibility is the operation of that network. The only thing you should do as a NOC is fix problems when they arise in the most expedient and professional manner possible. A customer wants to be comfortable, and assured that that will happen.

Things we never, EVER allow a customer to hear: 

Things we never, EVER allow a customer to hear I don’t know when it will be fixed. I can’t help you. I’m not qualified to answer that question. Let me transfer you to someone else.

It’s not always ideal. But -: 

It’s not always ideal. But - Where possible, the first person to take the call should be the person who follows it to completion. The customer hates being thrown between different engineers - but not having an answer is worse.

A Little About Systems: 

A Little About Systems Don’t just focus on ticketing. Make sure your people have information on everything they need to operate. If all the architects fell off the planet today, could you rebuild? It’s about agility.

Policies and Procedures: 

Policies and Procedures If you don’t write it down, it isn’t a procedure or a policy. There are no unwritten rules in a NOC. People cycle. Focus on procedures that affect the handling of a customer. The soft skills are usually the hardest to learn. Full disclosure? Troubleshooting Techniques

And a word on incidents.: 

And a word on incidents. Incidents, tickets, call it what you will. Whatever happens in the NOC: gets summarized to the account exec if the issue is of appropriate severity gets recorded - nothing EVER gets deleted from a customer log gets archived gets resolved - how much time will you spend on issues that never get fixed?

Some final notes - Conclusion: 

Some final notes - Conclusion It’s easy to throw a bunch of people in a room. It’s hard to build an efficient, productive, and well-oiled NOC (the same goes for networks) Record everything. Write everything down. Try to remain as flexible as possible. Look to your employees for cues on what their ideal environment should be. Remember your focus, and know your customer. Know what not to say.

Credits: 

Credits As far as the practice of NOC building goes, we must thank: herb@tomobiki.urusei.net - thanks for great private discussions on NOC building nanog subscribers - for talking about operational stuff

And finally: 

And finally Sean Donelan - for giving network operations folks an impetus for existing (due to his backhoe skills) Shouts to EFNet IRC #nanog: you know who you are

Part III: Evaluation Criteria: 

Part III: Evaluation Criteria Social engineer your own NOC. We like to rate NOC folks on: Do they sound professional? Do they have the answers? Do they follow us all the way through to resolution? Can they tackle difficult as well as simple problems?

Part III: Evaluation Criteria (cont).: 

Part III: Evaluation Criteria (cont). How many calls a day does your NOC take? Do you survey your customers, Cisco-TAC style? The three ‘P’s: Perception Passion (Emotion) Presentation

Evaluation Criteria: 

Evaluation Criteria They (customer) must perceive you as the people that will help. You (NOC) must present your solution with total assurance that you can handle the problem. You (NOC) must have passion in dealing with the customer. Make it friendly, even if it’s a network down emergency. It will come back to pay off.

Remember.: 

Remember. Your goal is to be professional, calm, cool and collected, even in the face of serious adversity. If you don’t have an answer, you know where to get it and how long it will take. You have to communicate to the customer and keep the customer’s viewpoint in mind, no matter what. As a NOC manager/builder, you have to make sure your people have the right tools.