01 Main Steps to Gridify an Application

Main steps to gridify an Application Miklos KozlovszkyMTA SZTAKIm.kozlovszky@sztaki.hu: 

Main steps to gridify an Application Miklos Kozlovszky MTA SZTAKI m.kozlovszky@sztaki.hu

Outline: 

Outline This talk gives a high-level view of application development in Grids Contents Review of concepts: grids and grid applications Types of Grid applications Challenges to researchers who write applications General steps of application gridification Practical: Preparing and submitting a job based on a simple non-grid application. Acknowledgements Gergely Sipos, SZTAKI, Hungary Mike Mineter, University of Edinburgh, UK, “Application Development and Aspects of gLite 3.0”, 2006 Vladimir Dimitrov, IPP-BAS, Bulgaria, “Practical: Porting applications to the GILDA grid”, Introduction to Grid Computing, EGEE and Bulgarian Grid Initiatives Plovdiv, 2006 Douglas Thain, The University of Notre Dame GILDA team

What is a grid application?: 

Definition Software that interacts with grid services to achieve requirements that are specific to a particular VO or user. What is a grid application?

Grids: a foundation for e-Research: 

Grids: a foundation for e-Research Enabling a whole-system approach Collaborative research / engineering / public service …

The vital layer : 

The vital layer Where computer science meets the application communities! VO-specific developments built on higher-level tools and core services Makes Grid services useable by non-specialists Grids provide the compute and data storage resources Production grids provide these core services.

Slide6: 

Types of grid applications

Complexities of grid applications: 

Complexities of grid applications Simple jobs – submitted to WMS to run in batch mode Job invokes grid services To read & write files on SE Monitoring For outbound connectivity (interactive jobs) To manage metadata … Complex jobs An environment controls multiple jobs on users’ behalf High-level services Portals with workflow Software written for the VO (or by the user) …

Invocation of applications: 

Invocation of applications From the UI Command Line Interfaces / Scripts APIs Higher level tools From desktop Windows applications Use Grids without awareness of them! But gLite not (yet) fully supporting Windows (more info: http://jessica.trigrid.it/grid2win/) From portals For recurring tasks: “core grid services” as well as application layer Accessible from any browser Tailored to applications Different portal solutions, and wide range of capabilities. Second part of this course: P-GRADE Portal

Multi-Grid P-GRADE Portal: 

EGEE Grid e.g. VOCE UK NGS P-GRADE-Portal London Rome Athens The portal can be connected to multiple grids Multi-Grid P-GRADE Portal

Characteristics of VOs: 

Characteristics of VOs What is being shared? resources of storage and/or compute cycles software and/or data Distinct groups of developers and of users? Some VOs have distinct groups of developers and users… Biomedical applications used by clinicians,…. …. Some don’t Physics application developers who share data but write own analyses Effect: need to hide complexity from the 1st type of VOs expose functionality to 2nd type of Vos many security issues

Different Goals for App. Development: 

Different Goals for App. Development I need resources for my research I need richer functionality MPI, parametric sweeps,… Data and compute services together… I provide an application for (y)our research How!? Pre-install executables ? Hosting environment? Share data Use it via portal? We provide applications for (y)our research Also need: Coordination of development Standards … Engineering challenges increasing

Slide12: 

Challenges to researchers who write grid applications

Challenges: 

Challenges Research software is often Created for one user: the developer Familiarity makes it useable Short-term goals: Used until papers are written and then discarded Grid applications are often used by a VO Without support from developer In new contexts and workflows Grid application developers are In a research environment Yet their s/w must have: Stability Documentation Usability Extendability i.e. Production quality Need expertise in: software engineering application domain grid computing

Consequences: 

Consequences Team work! Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions. From research to production software: ~5 times the effort. “80% of the time for last 10% of the functionality & reliability” Standardisation is key For re-use, for dynamic configuration of services,.. Both for middleware and domain specific Need to follow a deliberate development process Waterfall? Rapid prototyping? Requirements engineering, design, implementation, validation, deployment Engaged with the user community

Slide15: 

More about gLite services

gLite Grid Middleware Services: 

gLite Grid Middleware Services API Access Workload Mgmt Services Computing Element Workload Management Metadata Catalog Data Management Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Application Monitoring Connectivity Accounting Auditing Job Provenance Package Manager CLI

More about gLite services: 

More about gLite services During today the focus is on: New functionality in gLite 3.0 Workload Management Accessing data on SEs Can have massive files, too big to copy How to access these? Management of metadata May have many thousands of files Need to access and re-use based on characteristics… more than by their logical file names. Monitoring of applications May be running many long jobs What’s happening?!

Workload Management System: 

Workload Management System Helps the user accessing computing resources resource brokering management of input and output management of complex workflows Support for MPI job even if the file system is not shared between CE and Worker Nodes (WN) – easy JDL extensions Web Service interface via WMProxy

WMProxy: 

WMProxy WMProxy is a SOAP Web service providing access to the Workload Management System (WMS) Job characteristics specified via JDL jobRegister create id map to local user and create job dir register to L&B return id to user input files transfer jobStart register sub-jobs to L&B map to local user and create sub-job dir’s unpack sub-job files deliver jobs to WM

Complex Workflows: 

Complex Workflows Direct Acyclic Graph (DAG) is a set of jobs where the input, output, or execution of one or more jobs depends on one or more other jobs A Collection is a group of jobs with no dependencies basically a collection of JDL’s A Parametric job is a job having one or more attributes in the JDL that vary their values according to parameters Using compound jobs it is possible to have one shot submission of a (possibly very large, up to thousands) group of jobs Submission time reduction Single call to WMProxy server Single Authentication and Authorization process Sharing of files between jobs Availability of both a single Job Id to manage the group as a whole and an Id for each single job in the group

Basic taskswhile Porting applications to the Grid: 

Basic tasks while Porting applications to the Grid Developing a non-grid application (or inheriting and updating an ancient one); Go/no-Go decision about gridification Is it suitable for the Grid environment? “Cost/profit” analysis / feasibility study Typical Questions Groups: Current structure of the application Dependencies of the application Available resources (manpower, knowledge, etc.) Requirements for the gridified application Expected impact of gridification Requirements for the grid infrastructure More info: Application Description Template http://www.lpds.sztaki.hu/gasuc/?m=4

Basic tasks while Porting applications to the Grid (contd.): 

Basic tasks while Porting applications to the Grid (contd.) Grid environment access Requesting Certificates / VO membership Accessing Grid environment Appropriate VO UI machine account for command line Portal GUI account; Executing, Testing and Debugging the application; Testing the non-grid application (debugging in Grid environment is a hard task), creating use cases for single (non-grid) runs; Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files; Submitting the job to the Grid as small-scale pilot application;

Basic tasks while Porting applications to the Grid (contd.): 

Basic tasks while Porting applications to the Grid (contd.) Executing, Testing and Debugging the pilot application; IF something goes wrong THEN GOTO 4; IF everything seems to work THEN increase the scale of the application (increase problem size, amount of used resources); Optimizing the grid application;

Porting legacy code applications: 

Code from the past, maintained because it works Often supports business critical functions Not Grid enabled Port them onto the Grid with minimum user effort What to do with legacy codes in service Grids? Bin them and reimplement them as grid services Reengineer them  who knows the source code? Porting legacy code applications

Porting legacy code applications with GEMLCA: 

GEMLCA – Grid Execution Management for Legacy Code Architecture To deploy legacy code applications as Grid services without reengineering the original code and minimal user effort To support the development and execution of “legacy code” grid service workflows To make these functions available from a Web Portal GEMLCA GEMLCA P-GRADE Portal Objectives Porting legacy code applications with GEMLCA

The GEMLCA-view of a legacy code: 

Any code that correspond to the following model can be exposed as Grid service by GEMLCA: Legacy code Input data (files, command line params, env. vars) Output data (files) Call shared libraries Query databases Communicate over the network Perform computation . . . The GEMLCA-view of a legacy code

Legacy Application example: 

Workflow to analyse road traffic Manhattan road network generator Traffic simulators Analyser Legacy Application example

Another Legacy Application example: 

Molecular Dynamics Study of Water Penetration in Staphylococcal Nuclease using CHARMm Analysis of several production runs with different parameters following a common heating and equilibrium phase Another Legacy Application example

Practical tools @ gridification: 

Mercury monitor to debug/optimize and visualize parallel jobs. both at the workflow and job levels Practical tools @ gridification On-Line Monitoring

Practical tools @ gridification (contd.): 

Practical tools @ gridification (contd.) Hiding Grid remote storage system from a legacy application Parrot a handy tool for attaching old programs to new storage systems EGEE (gLite module) Data Access: GFAL, LFN, GUID, SRM, RFIO, DCAP, and LFC does not require any special privileges, any recompiling to existing programs For example, an anonymous FTP service is made available to vi like so: parrot vi /anonftp/ftp.cs.wisc.edu/RoadMap Or example with gsiftp: $./parrot app_name /gsiftp/<gsiftp usr without gsiftp://> (More info: http://www.cse.nd.edu/~ccl/software/manuals/parrot.html)

Common problems and obstacles: 

Common problems and obstacles The candidate-applications for porting usually are huge and complex. Some of them use low-level network functions and/or parallel execution features of a specific non-grid environment. Usage of non-standard or proprietary communication protocols. The complete source code might not be available, might not be well documented or its “out-of-host” usage is restricted by a license agreement.

Common obstacles (continued): 

Common obstacles (continued) The application might be written in many different programming languages – C, C++, C#, Java, FORTRAN etc. or even mixture of them. Applications may depend on third-party libraries or executables which are not available by default on some Grid worker nodes. Some application features could cause unintentional violation of Grid Acceptable Use Policies (Grid AUP). Furthermore, the application can have hidden security weakness which will be very dangerous in case of remote Grid job execution.

Common obstacles (continued): 

Common obstacles (continued) Some applications are pre-compiled or optimized for using on a machine with particular processor(s) only – Intel, AMD, in 32-bit or 64-bit mode, etc. But the Grid is heterogeneous! The application may contain serious bugs, which have never been detected while running in a non-grid environment. Finally, the formal procedure for accepting a new application to be ported to a Grid for production or even experimental purpose is not simple. Therefore, the porting of an arbitrary application to Grid could be very long, difficult and expensive process!

GASUC Grid Application Support Centre : 

GASUC Grid Application Support Centre AIM of the Center Provides assistance to current and future grid users and grid application developers to port legacy algorithms and applications onto large-scale grid infrastructures. Creates a bridge between Academy/Industry and Grid Supports and develops Grid applications for Grid communities Disseminates knowledge of concepts and tools used on Globus Toolkit 2, Globus Toolkit 4, LCG-2 and gLite grid environments. Established in January 2007 Full EGEE support Announced officially in EGEE Follow up in EGEE3 Cost model For non-commercial applications: free http://www.lpds.sztaki.hu/gasuc/

Grid Application Development Support Model: 

Grid Application Development Support Model 8 steps support model Contact phase Pre-selection phase Analysis phase Planning phase Prototyping phase Testing phase Execution phase Dissemination and feedback phase

In the practical session: 

In the practical session The application called MatrixDemo will be ported and executed in SGDEMO grid environment. (The program is borrowed from the “EGEE summer school” at MTA SZTAKI, 2006.) MatrixDemo is written in C programming language SGDEMO environment (gLite based) is supporting C, so porting the C or C++ programs is easy … hopefully

MatrixDemo program: 

MatrixDemo program MatrixDemo program performs some matrix operations – inverting, multiplying, etc. Usage: MatrixDemo has command line interface which accepts several arguments. Starting the program without any argument will display a short help. Example: MatrixDemo I V This will Invert (I) the matrix defined in the file named INPUT1 and will store the result in the file OUTPUT with verbose details (V).

MatrixDemo program (continued): 

MatrixDemo program (continued) Prerequisites: File MatrixDemo.c – the source code of the program. Files INPUT1 and INPUT2 – they contain matrix data in the following text format: rows, columns, cell1, cell2, cell3 … Where rows is an integer representing the number of rows. columns represents number of columns, and cell1, cell2 etc. are the cells of the matrix, floating point numbers separated by commas (,). A standard C compiler and linker. In this case we will use GNU C (gcc) already installed. File MatrixDemo.jdl – a prepared JDL (Job Description Language) file.