Exchange ??????????? :Exchange ??????????? ??? ???? ????????


?? :?? ?????? Exchange ?????????? ???? ??


What’s Out There? :What’s Out There? Benchmarks ?? …????????????? Why? ???? ???????? ??????? ???????????? ??????????,????????????


????? :????? ?????? LoadSim 2003, JetStress, ESP Sizing tools Varying levels Risk of GIGO as solution grows White papers KB Articles ????,?????


6???????? :6???????? ?????? ????SLA??????????????? ????? benchmarks ?????? ???? ?????? ??,??????


????: ???? :????: ???? User profiles ????????? IOPS/u Megacycles / mailbox Network bytes / mailbox ?????????


???? :???? ???? Server’s reaction to load Only part of the picture ??? Messages/sec Transactions/sec ?????? High-demand resource Workload-dependent Never really eliminate; just move


?????? :?????? CPU Memory Physical and Virtual I/O Disks Network Scale up or scale out?


?? :?? ?????? Exchange ?????????? ????? ????????? I/O ?? ?? ???? ??


Exchange 2003 ????? :Exchange 2003 ????? ?????????? ??????? ????? ??? Inbound SMTP gateways / SMTP Bridgehead X.400 / legacy connector servers DL Expansion servers ????? Free/busy ??(e.g., OAB Generator)


????? :????? ??????????: Blade serversFE servers, BE servers with SAN/iSCSIDense serverHighly available, scalable BE, FE serversHigh-capacity serversBridgehead servers, replicated sites/departmentalVirtual serversLab & hardware-independent QA ??: Power consumption Direct-attached versus network storage High-performance servers High-availability options


Exchange??: CPU :Exchange??: CPU ????: ???4 processors Family-dependent Yesterday: Pentium III Xeon Today: Xeon / Xeon MP / Opteron Goal: Burn 80% (a challenge?) ?????? Hyperthreading / dual-core Clock frequency (>3.6 GHz) L2, L3 cache (up to 8 MB L3 cache) Front-side bus / HyperTransport x64 extensions


????? :????? Time


Frequency, L3 Cache, FSB Comparisons :Frequency, L3 Cache, FSB Comparisons Front-Side Bus 400MHz ? 533MHz Clock speed boost without L3 cache Addition of L3 cache (1MB) Clock speed boost with L3 cache Platform: ProLiant DL360G3; 2P, Exchange 2003 RTM; identical configuration Workload: 3,000 MMB3. Response Time relatively consistent


Hyper-Threading Results :Hyper-Threading Results Platform: ProLiant DL580G2; 4P 2.0GHz/2M cache, Exchange 2003 RTM Workload: MMB3 15–25% boost


Exchange??: ?? :Exchange??: ?? ????????? 4 GB on high-end back-end servers (2,000 mailboxes and above) /3GB (BE and PF servers) with > 1 GB ????????????? HeapDecommitFreeBlockThreshold Watch Free System Page Table Entries: /USERVA:3030 ESE Cache tuning ??? FE, SMTP??? Check KBs 815372, 810371


Exchange??: ?? :Exchange??: ?? ????? Typically, 100 Mbit full duplex sufficient Consider Gigabit and NIC teaming if: Network backup/restore iSCSI or NAS storage – TCP Offload Engine support High concentration of OWA / POP / IMAP users Consider NICs with IPSec offload MAPI compression/buffer packing 30 – 40% improvement in cached mode synchronization, standard MAPI traffic over E2K ????? Dual switched 100 Mbit or Gbit default


Exchange??: ??????? :Exchange??: ??????? Size for I/O *and* capacity ?? SLA Architect for high availability, backup…don’t “add-on” VSS or replication ?????log?????? Performance and disaster recovery considerations RAID10 everywhere ???? Database: fast reads (<20ms) Logs: fast writes (<10ms) Bursty random I/O patterns Design for controller cache avoidance Design for Monday morning peak


Disk IOPs :Disk IOPs IO per-spindle dependencies RAID type Latency requirements Access pattern Read/write mix Spindle performance (rotational speed) ???? (FC and U320 SCSI) 10K disks: 120 IOPS per disk @ 20ms 15K disks: 160 IOPS per disk @ 20ms Conservative use of 100 IOPS per disk for design allows room for peak periods Figure are RAID1+0; RAID5 IOPS drops to ~90/ 10K disk Transaction logs require ~10% of SG IOPS


?????? :?????? MSA1000; 16 disks/database; 1 DB/LUN/RAID 0+1 Array; increasing load to saturation generated by JetStress ~20ms/write ~10ms/read


????: ????? :????: ????? SMTP Create a single partition; ~30 small msgs/sec/spindle RAID 0+1 recommended; 100% write-back cache X.400 and legacy connectors Separate MTA (RAID 5), logs (RAID 1) and page file, if possible Other FE servers Not I/O-intensive


????: ????? :????: ????? ?????????… Array / LUN design Journaling / Archiving VSS-based plexes (snapshot, clones) Temp/TMP drive Partition Alignment with physical sectors (Diskpar/Diskpart with SP1) …to mention a few


Database IOPs/u Rates :Database IOPs/u Rates Rough estimates (MAPI corporate users) ISP / “light” user: <.3 IOPs/u Corporate: .7 – 1.2 IOPs/u Microsoft: .9 – 3 IOPs/u Rates do not stay constant as load, mailbox sizes increase


RAID ???? :RAID ???? 34 disks for databases One large RAID 1+0 set Mostly 3-disk RAID5 sets Mostly 4-disk RAID6 (ADG) sets RAID5 performance bestwith multiple small R5sets


Storage Planning Calculator for Exchange :Storage Planning Calculator for Exchange


Controller Cache Mix :Controller Cache Mix RAID 1+0 Controller-specific 50/50 best Some write-cache necessary RAID 5 100% write is best Write gathering benefits Read-ahead detrimental to random workload Watermark setting recommended at 80%


HBA ?? :HBA ?? Queue depth is array- and deployment-specific Values for an HP EVA5000, Emulex HBAs Emulex HBA parameter Baseline Mod QueueTarget 1 0 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lpxnds\Parameters\Device0\ Number of Requests 50 150 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lpxnds\Parameters\Device\


NAS ? iSCSI ?? :NAS ? iSCSI ?? NAS ?????????????? ?????? Low – to 250, Medium – to 750, High – to 1,500 iSCSI Tested Scenarios for Windows Storage Servers Low – tested to 500 Medium to High – tested to 5,000 Expansion storage e.g., augmenting Fibre Channel Recovery Storage Group Public Folder Server Archiving (ILM- Information Lifecycle Management)


?? iSCSI ??…????????… :?? iSCSI ??…????????… Start with # & speed of spindles… Teamed GigE can handle upwards of 8,500 IOPS Approx: ~70 10K or ~50 15K disks Bottlenecked at Target Disk performance (e.g. move 10K?15K or ATA?SCSI) To/from targets: Increase bandwidth (esp. backups) Isolate GigE network NIC Teaming Additional NICs Jumbo Frames? Bottlenecked at Initiator iSCSI HBA (and TCP Offload Engine) instead of soft


Exchange??: ?? :Exchange??: ?? ??CPU??????????? ?????/ ????? ?? / ?? ?? Exchange ?? ??OAB ???? ?????? Mobile device support E.g., 1 BlackBerry user = impact of 2.21 MAPI users (CPU, storage and network) on Exchange BE server (based on lab testing of BES 3.5)


??????? :??????? Reference configuration included management, backup agents AV solution utilized STORE AV API


????? :????? ??????,???????????? ????????PF???????????PF ??? ????????‘ ?????? ??????? ???PF????????????? ?????????????? ????? (sort, search, expand)???? Hotfix to address existing deep hierarchies with different permissions at various nodes in the tree


????????? :????????? Maximum 250 subfolders / folder Consider ACL schema – arrange based on user sets that gain access to parent folder Maximum 10K items / folder Configure expiration accordingly, or segmenting into sub-topics Maximum 10K users / PF database VM issues Lab tests – STORE > 2.2 GB virtual bytes in clustered environment with no active mail users Monitor MSExchangeIS Public | Client Logons and Peak Client Logons Assume 1.6x scaling factor, and 1:2 R:W mix


Active Directory???? :Active Directory???? Typical configuration 2 P+, 2 GB+, RAID10 for DIT, RAID1 for logs Depends on replication, type of deployment Third party AD apps will impact Use /3GB on GC with > 1 GB RAM Larger ESE Cache means 20%– 40% disk I/O decrease Dedicated AD site for Exchange How many clients per Global Catalog Server? 1:4 GC/Exchange processor ratio GC co-located with mailbox servers Login times, DG decomposition, Delegate access GC with large user populations 4,000 clients per GC Use ClosestGC registry key for Outlook 2003 clients


Maintenance Considerations :Maintenance Considerations IS Online Maintenance Checking Active Directory for deleted mailboxes Minimal BE server impact Scheduling important to minimize AD impact Disk-intensive maintenance activities Permanently remove mailboxes / messages older than retention policy Online defrag Stagger intervals with backup schedule PF Online Maintenance Allow for 4 hours/day, and full online maintenance weekly


?? :?? ?????? Exchange ?????????? ???? ????????? ??


?????? :?????? ??????? ???? ???? ????????????? 95th percentile instead of average ????????


??????????? :???????????


???????? :????????


????? :?????


????? :?????


Topics :Topics ?????? Exchange ?????????? ???? ??


?? :?? Benchmarking: Caveat emptor. Develop own tests/baselines/acceptance criteria. ???? Exchange ?????????? ???????????? ????? CPU: 2 – 4. Consider fastest FSB, larger L3 cache over CPU speed. Memory: 4 GB Network: 100 Mbit; dual 100 Mbit for FE Active Directory 1 GC near each Exchange server and 4,000 users ??????????,?????????SLA?? ?????????????


???? :???? ???: ?????? Cache misses & small random I/O to the disks ~70/30 read/write ratio, 60/40 for sizing Transaction logging: ????? Small sequential I/O to the logs ?? I/O per second ????,????? ?? RAID ?? (from 5 to 1) ????(ms) ??HBAs


?? :??


???? :????


Slide 48:© 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.


Storage Sizing Exercise :Storage Sizing Exercise “Back of the Napkin” Three scenarios 4,000 “medium” users 8,000 “light” users 4,000 “heavier” users Baseline assumptions Baseline disk: 72.8 GB disk = 66 GB useable storage 10K spindle = 120 random IOPS 15K spindle = 150 random IOPS


Back of the Napkin4,000 “medium” users :0.8 IOPS average; 1.0 IOPS peak 4,000 users * 1.0 IOPS = 4,000 IOPS 3:1 ratio: 3,000 read and 1,000 write IOPS Backend disk IOPS: RAID 10 calculation Total IOPS = 3,000 + 2 * 1,000 = 5,000 42 drives @ 10K RPM or 34 drives @ 15K Backend disk IOPS: RAID 5 calculation Total IOPS = 3,000 + 4 * 1,000 = 7,000 59 drives @ 10K or 47 drives @ 15K 200 MB mailboxes 4,000 users * 200 MB * 1.5 (“fluff factor” for deleted items, maintenance, etc.) = 1,172 GB 1,172 GB ÷ 66 GB/drive = 18 drives Backend disks: RAID 10 calculation 18 drives * 2 (RAID 10) = 36 drives Best match to 36 drives 73GB @ 15K RPM (capacity) Backend disks: RAID 5 calculation 18 drives * 1.2 (RAID 5) = 22 drives Poor match to 59 (10K) or 47 (15K) IOPS Back of the Napkin4,000 “medium” users Capacity


Back of the Napkin8,000 “light” users :0.3 IOPS average; 0.5 IOPS peak 8,000 users * 0.5 IOPS = 4,000 IOPS 3:1 ratio: 3,000 read and 1,000 write IOPS Backend disk IOPS: RAID 10 calculation Total IOPS = 3,000 + 2 * 1,000 = 5,000 42 drives @ 10K RPM or 34 drives @ 15K Backend disk IOPS: RAID 5 calculation Total IOPS = 3,000 + 4 * 1,000 = 7,000 59 drives @ 10K or 47 drives @ 15K 200 MB mailboxes 8,000 users * 200 MB * 1.5 = 2,344 GB 2,344 GB ÷ 66 GB/drive = 36 drives Backend disks: RAID 10 calculation 36 drives * 2 (RAID 10) = 72 drives @ 73 GB Would match 36 drives 146 GB @ 15K RPM or 42 drives 146GB @ 10K RPM (I/O) Backend disks: RAID 5 calculation 36 drives * 1.2 (RAID 5) = 44 drives @ 73 GB Would match 47 drives 73 GB @ 15K (I/O) IOPS Capacity Back of the Napkin8,000 “light” users


Back of the Napkin4,000 “heavier” users :1.2 IOPS average; 1.5 IOPS peak 4,000 users * 1.5 IOPS = 6,000 IOPS 3:1 ratio: 4,500 read and 1,500 write IOPS Backend disk IOPS: RAID 10 calculation Total IOPS = 4,500 + 2 * 1,500 = 7,500 63 drives @ 10K RPM or 50 drives @ 15K Backend disk IOPS: RAID 5 calculation Total IOPS = 4,500 + 4 * 1,500 = 10,500 88 drives @ 10K or 70 drives @ 15K 400 MB mailboxes = “heavier” IO profile 4,000 users * 400 MB * 1.5 = 2,344 GB 2,344 GB ÷ 66 GB/drive = 36 drives Backend disks: RAID 10 calculation 36 drives * 2 (RAID 10) = 72 drives @ 73 GB Best match 50 drives 146 GB @ 15K RPM (I/O) Backend disks: RAID 5 calculation 36 drives * 1.2 (RAID 5) = 44 drives @ 73 GB Poor match to 88 (10K) or 70 (15K) drives IOPS Capacity Back of the Napkin4,000 “heavier” users


Exchange Best Practices Analyzer :Exchange Best Practices Analyzer A health check tool Checks current configuration against “best practices” Only as good as the rules or current “best practices” Contains update capability to incorporate new data and best practices Update can be done each time the tool is run Rules information held in configuration files Available as web download as part of Exchange “All-in-1 tools” or www.expba.com or www.microsoft.com/exchange/exbpa