AHE-Steady_State-KT

Slide 1: 

IGA AHE Steady State Marina Soubhia

Agenda : 

2 Agenda 1.Environment 1.1 Account Standards 1.2 DPROP Standards 1.3. POD accounts 2. Documentation 2.1. Applications Information 2.2. Access 2.3. How To Open a PMR 2.4. Problem Tracking Issues Log 2.5. Steady State Checklist 2.6. Access Matrix 2.7. Team Info

Agenda : 

3 Agenda 3. Proccesses 3.1. Problem/Change Guidelines for DBA Team 3.2. How To Prepare an Off-Shift Request 3.3. How to Close a Problem Record 3.4. Weekend Oncall 4. SME Responsabilities 4.1. Health Check 4.2. Inbox Monitoring 4.3. Password Change

1.Environment : 

4 1.Environment Our servers are divided into two different kind of servers: Green Zone – Internal server with internet access Ex: g01edrdb011.ahe.pok.ibm.com g03edrdb004.ahe.boulder.ibm.com g03edrdb002.ahe.boulder.ibm.com Blue Zone – Internal server without internet access Ex: b03aedrdb001.ahe.boulder.ibm.com b03edrdb077.ahe.boulder.ibm.com b01zciapp003.pok.ibm.com b03aedrdb002i.ahe.boulder.ibm.com (IVT)

1.Environment : 

5 1.Environment 1.1 Account Standards AHE Accounts have some standards for reports, scripts, filesystems, etc. /home/$INSTANCE/sqllib/db2diag – Diag.log path (V8) /db/$INSTANCE/db2diag – Diag_log path (V9) – If this is not set up correctly you should check to fix that. /db/$INSTANCE/db2data – Where tablespaces are located (database installed) /db/$INSTANCE/db2backup – Where backup images are sent to /db/$INSTANCE/actlog – Where transaction logs are located /db/dbawork/dbtools – Where scripts are located /db/dbawork/db2reports – Where reports for the scripts are located

1.Environment : 

6 1.Environment 1.1 Account Standards Weekly Scripts (Maintanence Window): Maint Wrapper (Offline backup, Reorgchk, Reorg, Runstats and Rebind) Db2look Get db cfg Get dbm cfg Daily Scripts: Online backup Check tablespace Check disk DB Activate Runstats Reorgchk Diag Compress Transaction logs cleanup on TSM

1.Environment : 

7 1.Environment 1.3 POD Accounts POD accounts are the critical accounts that require special attention. The SMEs from these accounts are responsible for checking CRs, Offshift requests, etc. Blue Horizon: SMEs -> Marina (msoubhia@br) and Maira (mtmelo@br) CCE: SMEs -> Marina (msoubhia@br) and Maira (mtmelo@br) CPE: SMEs -> Marina (msoubhia@br) and Maira (mtmelo@br) ePricer: SME -> Karina (karinar@br) ICC: SME -> Luciano (lschaper@br). Backup: Leonardo (leotoras@br) PROFIT: SMEs -> Luciano (lschaper@br) Backup: Leonardo (leotoras@br) BRIO: SMEs -> Luciano (lschaper@br) Backup: Leonardo (leotoras@br) RPM: SME -> Venkat (nrvenkat@us) NOTE: Only ePricer has a dedicated DBA so only Karina can work on tickets/CRs for this account. For the another ones anyone can work on tickets/CRs for them.

1.Environment : 

8 1.Environment 1.2 DPROP Standards We have separated IDs for Apply/Capture (We can have both on the same ID) On DPROP ID crontab we can find: Stop Apply/Capture script before Maintanence Window Start Apply/Capture script after Maintanence Window Apply Check (To make sure apply is running ok) Capture Lag (To make sure capture is running ok) Logs and scripts have special directories for them /home/$INSTANCE/torun – New Apply/Capture logs home/$INSTANCE/torun/logs – Old Apply/Capture logs /home/$INSTANCE/scritps – DPROP scripts

2. Documentation : 

9 2. Documentation All of our documenation is located on ADD Notes:///87256F3C00576FAF/ Notes:///87256F18007B8D77/ On ADD Support Docs you will find all documents owned by DB2 Team On ADD you will find all needed information for all accounts. Including SDM name, Support queues on manage now, etc.

2. Documentation : 

10 2. Documentation 2.1. Application Information If you need to know specific information for some application on DB2 side like Password Change, HADR/HACMP information, etc, you should check on Application Information on ADD Support Docs (Secured Docs – Documents) You’ll find there all Projects supported by us and some specific information about them.

2. Documentation : 

11 2. Documentation 2.2. Access To work on AHE environment there are some access you need to request: ADD (Send an email to the team leads (mvyazmen@us.ibm.com, carlane@us.ibm.com) requesting access to this database) GSA Id (https://daatmaster.austin.ibm.com/cgi-bin/ndaat/daat) Manage Now 4 (Notes:///87256F3C00576FAF/FA5A795619612F3287257398006263DA/11D2E5AE4B05E51487257371004F9F07) WRT (Send an email to the team leads (mvyazmen@us.ibm.com, carlane@us.ibm.com) requesting access to this database) ICC YM and Profit (https://d02opscl001.southbury.ibm.com/UIDM/) DBA Team and POD Inbox (Contact Mike requesting the access) Sametime BlueGroup (Contact Renato (renatoa@br.ibm.com) requesting the access) Here are only listed the websites you should request access. For more detailed instructions about how to use them please access the New Team Members Information on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Documentation - New Team Members Information

2. Documentation : 

12 2. Documentation 2.3. How To Open a PMR When you have any issue you need help from second level you should open a PMR. There are two ways to open it: Using the ibm.com website (https://www-304.ibm.com/support/electronic/portal/uprtransition.wss?category=2&locale=en_us) Calling 800-426-7378 --> Options : 2,2,6 (from Brazil dial 0001118004267378) You’ll need to have the customer numbers. For our team you need to use 0519775 or 6759977 (These 2 customer numbers have extended support for v8 until Dec 2010.) For more detailed information please check on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Documentation – How To Open a PMR

2. Documentation : 

13 2. Documentation 2.4. Problem Tracking Issues Log There are two kind of issues you need to report when they happen so a better RCA can be done: High CPU and Filesystem Full issues. This is reported on: ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Problem Tracking Issues Log High CPU Issues Log: You’ll need to fill the document with Problem ticket number, Server name, Assignee of the ticket, Date of occurance and some comments about the issue. Note: Before working on any CPU issue please check if this is not documented already. If it is, you can have more information about that.

2. Documentation : 

14 2. Documentation 2.4. Problem Tracking Issues Log Filesystem Issue Tracking You’ll need to fill the document with Date of occurance, DBA Name, Problem Number, Severity number, application name, Server, Filesystem, Issue, Cause of the issue, How was the issue fixed?, Is the problem likely to come back? Why?. What else needs to be done beyond the resolution to make sure the problem does not come back?, Other comments, DPE Intervention was needed? Note: Before working on any Filesystem issue please check if this is not documented already. If it is, you can have more information about that.

2. Documentation : 

15 2. Documentation 2.5. Steady State Checklist When a new project comes to Steady State team we receive a WRT with a Steady State Checklist that needs to be completed to make sure everything is working correctly. You’ll need to check passwords, scripts, TSM, etc. After everything is checked a note needs to be sent to the Transition person who worked on the Projetct Deploy copying asddbapc@us. If there’s something to correct, the DBA from Transition team needs to fix that. This checklist is on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – STEADY STATE TEAM – Final Verification Checklist: Account Moving to Steady State Support)

2. Documentation : 

16 2. Documentation 2.6. Access Matrix This document details the guidelines for granting database access on Blue and Green zone servers. This can be found on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Security - Database Security Procedures : Database Access Checklist

2. Documentation : 

17 2. Documentation 2.7. Team Info On our team info we can find the servers/instances/databases we support, manage now queues, some useful commands, passwords, DBA list, Sunset servers, etc. You’ll find the lasted version on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – TEAM_INFO – DBA Team_info

3. Processes : 

18 3. Processes 3.1. Problem/Change Guidelines for DBA Team There are some things you can’t do using a problem ticket or a problem ticket without SDM approval on it. If you need to know what can/can’t be done, you can check this document. It’s located on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Documentation – Problem/Change Guidelines for DBA Team NOTE: Web Identity application has blank approval (No SDM approval is required on Problem tickets, only on CRs.)

3. Processes : 

19 3. Processes 3.2. How To Prepare an Off-Shift Request If you are assigned to a Change Record that is not on your shift you need to create an offshift request and send it to mvyazmen@us.ibm.com and asddba00@us.ibm.com. On this request you need to add detailed instructions for the CR. If you don’t know how this should be created, you can check the doc on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Documentation – How To Prepare an Off-Shift Request

3. Processes : 

20 3. Processes 3.3. How to Close a Problem Record There are some steps you need to follow to close your problem ticket on Manage Now. Duration: The duration of your problem ticket must be how much time your ticket was on OPEN state. For example: If your ticket was opened on 02/22/2010 12:00 and you’re closing it on 02/25/2010 12:00 the duration is 3 days. Even if you spent only 10 minutes fixing the issue. NOTE 1: If your ticket was on HOLD for 1 day, you need to calculate the duration: 3 days – 1 day = 2 days of duration. NOTE 2: If your ticket is on HOLD you must reopen it before closing. NOTE 3: If this was a false alert Resolved Time should be equal to Closed time and Duration should be ZERO.

3. Processes : 

21 3. Processes 3.4. Weekend Oncall When you’re the Oncall DBA during the weekend you must: Contact the asddba00@us.ibm.com functional id asking if there’s any handover. If the answer is yes, you need to ask them to send a note with detailed information about the ticket/change. After that, you can say you will join the chat and logon using asddba id. Setup the pager so we can avoid customer impact. The document that has all the steps to get this set can be found on ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Hot Pager - Switching AHE DB2 UDB DBA Hot Page Check dispatcher queue (ANDREC) on Manage Now for problem ticket that will be expired by Monday;

4. SME Responsabilities : 

22 4. SME Responsabilities 4.1. Health Check Every quarter we need to run the health check on the accounts we are SME: Check if HC is using the newest version Check if db2shc already ran by crontab. If it didn’t, run it manually Check for violations and update the health check spreadsheet. The document for Health Check procedures can be found on: ADD Support Docs – Secured Documents – Documents – DBA/DB2 – Security - Database Security Procedures : Security Health Checking for Quarterly Reviews & New Deployments Note: This document has all the steps to fix the violation. But in our case, Leonardo Toras (leotoras@br) is the responsible for open the CRs after checking the spreadsheet.

4. SME Responsabilities : 

23 4. SME Responsabilities 4.1. Inbox Monitoring We are also responsible for monitoring the asddba00 inbox (Notes:///8725727A006FD8C7/) or the asddbap1 [POD] (Notes:///8725729900271A43/). These are the steps we need to take: Check the alerts for your servers on the inbox Open a WRT or a Problem Ticket on Manage Now (Urgent issues) Assign it to the dispatcher. If it’s a WRT this should be assigned to DB2 UDB Internal Dispatcher/Brazil/Contr/IBM. If it’s a problem ticket this should be transferred to NUS_W_SSAHEDBA queue.

4. SME Responsabilities : 

24 4. SME Responsabilities 4.3. Password Change Our passwords expire every 90 days and we have a schedule CR to change them. This CR has 365 days of duration and inside of it you can see the dates that the passwords will be changed. Ex: CR# 1421460. PS: All of the ids uses the SAME password except admin ids (fenc, db2das, etc [These are changed by Mike.]). Steps: On db2das home, you’ll find the pull_new.ksh script, that will generate the .password profile for your server. You need to run so the standard script will run during the password change and update the password automatically, reading this .password file. On the password change day, you need to check one by one to see if the script worked ok and changed the passwords, specially for DPROP ids.

Slide 25: 

25 25 Thank You !