haas 20041025

Parallel File Systems: 

Parallel File Systems Peter W. Haas haas@hlrs.de Universität Stuttgart Höchstleistungsrechenzentrum Stuttgart (HLRS) www.hlrs.de

Table of Contents: 

Table of Contents Global Parallel File System Developments ASCI PathForward File System Strategy Lustre, Panasas ActiveScale, IBM SAN FS Parallel File Systems w/o Metadata Servers 2. Rationale of HSM Systems Storage Design Space Future of Tape Scalable Global Parallel HSM Systems 3. Layering of Legacy Networks IEEE 10Gbase-* Standardization Backplane and Data Center Networks 4. Conclusion

Unified Heterogeneous File Systems – for 100+ years: 

Unified Heterogeneous File Systems – for 100+ years Shannon Filing Cabinet, Schlicht & Field Co., Rochester, NY, 1886 http://www.officemuseum.com

SGS File System SOW: Figure 2: 

SGS File System SOW: Figure 2

SGS File System SOW: Figure 4: 

SGS File System SOW: Figure 4

SNIA Storage Model: 

SNIA Storage Model File/record subsystem Database (dbms) File system (VFS) Storage domain Application Application Services subsystem Services subsystem Discovery, monitoring Resource mgmt, configuration Security, billing Redundancy mgmt (backup, …) High availability (fail-over, …) Block subsystem Storage devices (disks, …) Block aggregation Device-based block aggregation Device-based block aggregation SN-based block aggregation SN-based block aggregation Host-based block aggregation Host-based block aggregation Copyright 2000, Storage Network Industry Association Parallel Data Management APIs

Slide7: 

File Systems for Clusters File Systems for Clusters

Slide8: 

Lustre Solution Lustre Solution

Slide9: 

Access Control Data Transfers Coherence Management Storage Management Clients Object Storage Targets Metadata Control Security & Resource Database High-Performance, Scalable, Open Distributed File System for Clusters and Shared-Data Environments Linux cluster = Lustre is a completely new start Scalability at inception Separation of Meta data & file data Scalable meta data Scalable file data Block management at OST level Efficient locking Object architecture Lustre is open source Check www.lustre.org Not encumbered by existing architecture

FS Requirements for HPTC: 

FS Requirements for HPTC Philippe Bernadat: The Lustre File System 3rd HLRS Workshop on SGPFS, March 22, 2004

Lustre Object Storage Model: 

Lustre Object Storage Model

Lustre Components & Interactions: 

Lustre Components & Interactions LDAP Server Lustre Client (CFS/OSC) Meta-data Server (MDS) recovery, file status, file creation file I/O & file locking Configuration information, network connection & security management Directory operations, Meta-data & concurrency

Bringing it all together: 

Bringing it all together Object-Based Disk Server (OBD) Lock Server Directory Metadata & Concurrency MDS Client FS Metadata WB cache OSCs MDC Lock Client Networking Recovery, File Status, File Creation OST Ext3, Reiser, XFS, … FS CFS LOV Networking Load balancing Recovery Lock Client Lock Server MDS Server Recovery SAN, Fibre Channel, … Device (Elan,TCP,…) Portal NAL’s Portal Library NIO API Request Processing Ext3, Reiser, XFS, … FS Networking Recovery SAN, Fibre Channel, … System & Parallel File I/O, File Locking Network Fabric

Recent Results: 

Recent Results File I/O % of raw bandwidth >90% Achieved OST throughput 270 MB/s Achieved client I/O 260 MB/s GigE end-to-end throughput 120 MB/s Aggregate I/O 1,000 clients 11.1 GB/s Attribute retrieval rate in 10M file directory, 1,000 clients 7,500/s Creation rate one directory, 1,000 clients 5,000/s * Performance measurements made in November 2003 on production clusters at Lawrence Livermore National Laboratories and the National Center for Supercomputing Applications.

Clusters Demand a New Storage Architecture: 

Clusters Demand a New Storage Architecture Garth Gibson: ActiveScale Storage Cluster 3rd HLRS Workshop on SGPFS, March 22, 2004

New Object Storage Architecture: 

New Object Storage Architecture

Object Access Example: 

Object Access Example

Object Storage Bandwidth: 

Object Storage Bandwidth

How does an Object Scale ?: 

How does an Object Scale ?

Additional Strengths of Object Model: 

Additional Strengths of Object Model

Object Storage Access Security: 

Object Storage Access Security

Standardization Timeline: 

Standardization Timeline

ActiveScale SW Architecture: 

ActiveScale SW Architecture

Storage Tank (SAN FS): 

Storage Tank (SAN FS) David Pease: Storage Tank Research Directions 3rd HLRS Workshop on SGPFS, March 22, 2004

Storage Tank Overview: 

Storage Tank Overview Storage Tank is a distributed file system that provides: High-performance, heterogeneous file sharing using Storage Area Network (SAN) technology Automated, policy-based storage and data management facilities (like those provided on mainframes) Virtually unlimited SAN-wide scalability Enterprise-level availability IBM TotalStorage SAN File System (SAN FS)

Storage Tank Overview: 

Storage Tank Overview

Object-based Storage: 

Object-based Storage Concept Storage (controller or disk) manages “objects” One file maps (roughly) to one object File System does I/O to objects, rather than blocks Security Object has security key Key is shared between metadata server and storage device MDS gives non-forgeable credential to client All allocation and I/O done under credential Scalability space allocation offloaded to storage subsytem Copy Services potential for more intelligent copy services

Storage Tank Object Store Extensions: 

Storage Tank Object Store Extensions

Distributed/Grid Tank: 

Distributed/Grid Tank WAN/LAN inter-tank protocol NAS Existing data stored in NAS can be accessed through DST. Legacy NAS hardware investment is protected. Can be extended to support smooth migration of data from NAS to ST. GridFTP NAS protocols clients always access files from its local ST cluster Cache and Replica Management using DFMAPI SAN LAN MDS Cluster w/ DST Features DST caches/replicates heterogeneous/remote data locally Host (ST Client) Local Tank Grid-based Storage DST allows Data Grids to be accessed through a local Tank cluster utilizing Grid protocols and services to resolve file names and file locations. Remote Tank Remote Tank clusters accessed through a local Tank cluster, providing local SAN performance and single site semantics. Virtual Filesystem Directory Service Replica Location Service file name and location resolution

SGS File System SOW: Table 5: 

SGS File System SOW: Table 5

Advanced Areal Density Trends: 

Advanced Areal Density Trends M. Leonhardt 4-9-02

Storage Cost: 

Technology Vs Economics Storage Cost

HPSS Architecture: 

HPSS Architecture Shared, secure global file system Metadata-mediated via database based on IBM DB2 Highly distributed Multiple movers and subsystems for scalability Parallel disk and tape I/O API for maximum control and performance Parallel FTP (pftp) Proven Petabyte capability with no identified upper limits Joint IBM and DOE sponsorship assures both COTS and large (30 person) dev and test staff

Slide38: 

Detail Showing HPSS Parallel Data Movers with PLFM Capability SAN Disk Capability Or Capacity Platform PFTP Client 1 2 PLFMs Can Reside on any platform connected to SGFS open(), seek(), read() User interface Orchestrates transfer SAN Disk SAN Disk SAN Disk

Slide39: 

nMDS nO1 nO2 nOn . . . . MDS File Metadata File Location Data OST Configuration Block Allocation Data 1. mirrored nMDS with featured storage classes 2. nMDS becomes metadata store of its own Clnt Lustre native HSM Capability HSM Interconnect

Slide40: 

Lustre: Attachment of Industry standard HSMs O1 O2 On . . . . TSM1 TSM2 TSMn . . . . MDS File Metadata File Location Data OST Configuration Clnt Block Allocation Data Commercial HSM

IEEE 10GBase-* Standardization: 

IEEE 10GBase-* Standardization

Ethernet Backplane (Study Group): 

Ethernet Backplane (Study Group)

Data Center Ethernet (CFI)Facilitation of Ethernet Clustering: 

Data Center Ethernet (CFI) Facilitation of Ethernet Clustering

Data Center Ethernet (CFI): 

Data Center Ethernet (CFI)

Data Center Ethernet (CFI): 

Data Center Ethernet (CFI)

DWDM-GBIC: Optical Data Sheet: 

DWDM-GBIC: Optical Data Sheet 1: Measured with a 10-12 BER and with OSNR 20 dB @ 0.1 nm RBW 2: Measured with a 0.1 nm resolution bandwidth (RBW). 3: Equivalent to 200 km of G.652 fiber with 18 ps/nm*km of dispersion coefficient 4: Measured at 3600 ps/nm of dispersion and with OSNR 20 dB @ 0.1 nm RBW Product Numbers: DWDM-GBIC-XX.XX 32 different SKUs – one per wavelength

High Performance at its time: 

High Performance at its time Distance between towers: About 5 to 20 Km Bandwidth: 0.02 Baud Remark: Faster than a running slave 300 B Chr. Already: Wireless and COARSE WAVELENGTH Division Fires on towers have been used along the Dutch and Belgian coast to warn against the invasions of the Vikings + 800 - 1000 Arie van Praag - HNF-Europe a.van.praag@cern.ch

Conclusions on Parallel File and HSM Systems: 

Conclusions on Parallel File and HSM Systems Parallel File System Considerations Uniform Global Name Space No limitations wrt # of Servers, Clients, Storage subsystems Security is primary concern: Kerberos and/or PKI authentication Authorization via Access Control Lists: NFS V4, NTFS Parallel Metadata Servers required for HA and performance HSM Systems Either part of a Parallel File System, or attached via parallel Interface X/Open XDSM and XBSA specify interfaces between File Systems HSM should enable direct and partial access to files on any level HSM configuration should be based on an IP SAN Long-term repositories require HSMs with independent GNS

Slide49: 

For Details please refer to 3rd HLRS Workshop on SGPFS at URL: http://www.hlrs.de/news-events/events/2004/hwwws User Name: hww3gfs Password: paragon Thank You !