TrilinosOverviewAndF uturePlans

Trilinos Overview and Future PlansMichael A. HerouxSandia National Laboratories: 

Trilinos Overview and Future Plans Michael A. Heroux Sandia National Laboratories Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000.

Trilinos Development Team: 

Trilinos Development Team Chris Baker Developer of Anasazi, RBGen Ross Bartlett Lead Developer of MOOCHO, Stratimikos, RTOp, Thyra Developer of Rythmos Paul Boggs Developer of Thyra Erik Boman Lead Developer Isorropia Todd Coffey Lead Developer of Rythmos Jason Cross Developer of Jpetra David Day Developer of Komplex Clark Dohrmann Lead developer of CLAPS Michael Gee Developer of ML, Moertel, NOX Bob Heaphy Lead developer of Trilinos SQA Mike Heroux Trilinos Project Leader Lead Developer of Epetra, AztecOO, Kokkos, Komplex, IFPACK, Thyra, Tpetra Developer of Amesos, Belos, EpetraExt, Jpetra Ulrich Hetmaniuk Developer of Anasazi Robert Hoekstra Lead Developer of EpetraExt Developer of Epetra, Thyra, Tpetra Russell Hooper Developer of NOX Vicki Howle Lead Developer of Meros Developer of Belos and Thyra Jonathan Hu Developer of ML Sarah Knepper Developer of Komplex Tammy Kolda Lead Developer of NOX Joe Kotulski Lead Developer of Pliris Rich Lehoucq Developer of Anasazi and Belos Kevin Long Lead Developer of Thyra Developer of Belos and Teuchos Roger Pawlowski Lead Developer of NOX Michael Phenow Trilinos Webmaster Lead Developer of New_Package Developer WebTrilinos Eric Phipps Lead developer Sacado Developer of LOCA, NOX Dennis Ridzal Lead Developer of Aristos Marzio Sala Lead Developer of Didasko, Galeri, IFPACK, WebTrilinos Developer of ML, Amesos Andrew Salinger Lead Developer of LOCA, Capo Paul Sexton Developer of Epetra and Tpetra Bob Shuttleworth Developer of Meros. Chris Siefert Developer of ML Bill Spotz Lead Developer of PyTrilinos Developer of Epetra, New_Package Ken Stanley Lead Developer of Amesos and New_Package Heidi Thornquist Lead Developer of Anasazi, Belos, RBGen and Teuchos Ray Tuminaro Lead Developer of ML and Meros Jim Willenbring Developer of Epetra and New_Package. Trilinos library manager Alan Williams Lead Developer Isorropia Developer of Epetra, EpetraExt, AztecOO, Tpetra

Target Problems: PDES and more…: 

Target Problems: PDES and more… PDES Circuits Inhomogeneous Fluids And More…

Motivation For Trilinos: 

Motivation For Trilinos Sandia does LOTS of solver work. When I started at Sandia in May 1998: Aztec was a mature package. Used in many codes. FETI, PETSc, DSCPack, Spooles, ARPACK, DASPK, and many other codes were (and are) in use. New projects were underway or planned in multi-level preconditioners, eigensolvers, non-linear solvers, etc… The challenges: Little or no coordination was in place to: Efficiently reuse existing solver technology. Leverage new development across various projects. Support solver software processes. Provide consistent solver APIs for applications. ASCI was forming software quality assurance/engineering (SQA/SQE) requirements: Daunting requirements for any single solver effort to address alone.

Evolving Trilinos Solution: 

Evolving Trilinos Solution Trilinos1 is an evolving framework to address these challenges: Fundamental atomic unit is a package. Includes core set of vector, graph and matrix classes (Epetra/Tpetra packages). Provides a common abstract solver API (Thyra package). Provides a ready-made package infrastructure (new_package package): Source code management (cvs, bonsai). Build tools (autotools). Automated regression testing (queue directories within repository). Communication tools (mailman mail lists). Specifies requirements and suggested practices for package SQA. In general allows us to categorize efforts: Efforts best done at the Trilinos level (useful to most or all packages). Efforts best done at a package level (peculiar or important to a package). Allows package developers to focus only on things that are unique to their package. 1. Trilinos loose translation: “A string of pearls”

Trilinos Strategic Goals: 

Trilinos Strategic Goals Scalable Solvers: As problem size and processor counts increase, the cost of the solver will remain a nearly fixed percentage of the total solution time.   Hardened Solvers: Never fail unless problem essentially unsolvable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error. Full Vertical Coverage: Provide leading edge capabilities from basic linear algebra to transient and optimization solvers. Grand Universal Interoperability: All Trilinos packages will be interoperable, so that any combination of solver packages that makes sense algorithmically will be possible within Trilinos.  Universal Accessibility: All Trilinos capabilities will be available to users of major computing environments: C++, Fortran, Python, Web Universal Solver RAS: Trilinos will be: Integrated into every major application at Sandia (Availability). The leading edge hardened, efficient, scalable solutions for each of these applications (Reliability). Easy to maintain and upgrade within the application environment (Serviceability). Algorithmic Goals Software Goals

Trilinos Statistics: 

Trilinos Statistics Stats: Trilinos Download Page 11/01/2006.

One recent success story: 

One recent success story Marzio Sala, Peter Arbenz, ETH with biomedical institute of ETH. Linear elasticity solution of trabecular bones based on Trilinos, HDF5 and ParMETIS. The largest computation uses the following grid: # finite elements: 247'734'272 # grid vertices: 392'912'120 # matrix rows: 1'178'736'360 HDF5 file size: 37'798 Gbytes Multigrid matrix-free preconditioner. Less than 10 minutes 1024 CRAY XT 3 processors. Custom-developed matrix-free multigrid. Everything else written from scratch By a (talented) master thesis student in less than 6 months. Using Epetra/AztecOO/ML.

External Visibility: 

External Visibility Awards: R&D 100, HPC SW Challenge (04). www.cfd-online.com: 5 SciDAC Solicitations. Industry Collaborations: Boeing, Goodyear. Linux distros: Debian, Mandriva. Star-P Interface. SciDAC TOPS-2 partner. Nearly 3000 downloads since March 2005. Occasional unsolicited external endorsements such as the following two-person exchange on mathforum.org: > The consensus seems to be that OO has little, if anything, to offer > (except bloat) to numerical computing. I would completely disagree. A good example of using OO in numerics is Trilinos: http://software.sandia.gov/trilinos/

Trilinos Presentation Forums: 

Trilinos Presentation Forums ACTS “Hands-on” Tutorial: Past: Aug 22-24, 2006. Next: Aug 21-23, 2007. At Lawrence Berkeley Lab. Next Trilinos User Group Meeting: Date: Nov 6-8, 2007. (Elections) Oct 30-Nov 1, 2007. (Halloween) CSRI building. Other location? SciDAC opportunities (indirect and direct).

Trilinos Package Concepts: 

Trilinos Package Concepts Package: The Atomic Unit

Trilinos Packages: 

Trilinos Packages Trilinos is a collection of Packages. Each package is: Focused on important, state-of-the-art algorithms in its problem regime. Developed by a small team of domain experts. Self-contained: No explicit dependencies on any other software packages (with some special exceptions). Configurable/buildable/documented on its own. Sample packages: NOX, AztecOO, ML, IFPACK, Meros. Special package collections: Petra (Epetra, Tpetra, Jpetra): Concrete Data Objects Thyra: Abstract Conceptual Interfaces Teuchos: Common Tools. New_package: Jumpstart prototype.

Trilinos Package Summary: 

Basic Linear Algebra classes Block Krylov Methods Scalable Preconditioners 3rd Party Direct Solver Package. Abstract Interfaces. Trilinos Package Summary

Why Packages?: 

Why Packages?

Package Interoperability: 

Package Interoperability

Interoperability vs. Dependence (“Can Use”) (“Depends On”): 

Interoperability vs. Dependence (“Can Use”) (“Depends On”) Although most Trilinos packages have no explicit dependence, each package must interact with some other packages: NOX needs operator, vector and solver objects. AztecOO needs preconditioner, matrix, operator and vector objects. Interoperability is enabled at configure time. For example, NOX: --enable-nox-lapack compile NOX lapack interface libraries --enable-nox-epetra compile NOX epetra interface libraries --enable-nox-petsc compile NOX petsc interface libraries Trilinos configure script is vehicle for: Establishing interoperability of Trilinos components… Without compromising individual package autonomy. Trilinos offers seven basic interoperability mechanisms. ../configure –enable-python

Trilinos Interoperability Mechanisms(Acquired as Package Matures): 

Trilinos Interoperability Mechanisms (Acquired as Package Matures)

Interoperability Example: ML: 

Interoperability Example: ML ML: Multi-level Preconditioner Package. Primary Developers: Ray Tuminaro, Jonathan Hu, Marzio Sala. No explicit, essential dependence on other Trilinos packages. Uses abstract interfaces to matrix/operator objects. Has independent configure/build process (but can be invoked at Trilinos level). Interoperable with other Trilinos packages and other libraries: Accepts user data as Epetra matrices/vectors. Can use Epetra for internal matrices/vectors. IFPACK, Amesos, AztecOO, etc. objects as smoothers, coarse solvers. Can be used via Thyra abstract interfaces. PyTrilinos. Can be built via Trilinos configure/build process. Can be driven via Teuchos ParameterLists. Available as preconditioner to all other Trilinos packages. Available to PETSc users without dependence on any other Trilinos packages.

Package Maturation Process: 

Package Maturation Process Asynchronicity

Day 1 of Package Life: 

Day 1 of Package Life CVS: Each package is self-contained in Trilinos/package/ directory. Bugzilla: Each package has its own Bugzilla product. Bonsai: Each package is browsable via Bonsai interface. Mailman: Each Trilinos package, including Trilinos itself, has four mail lists: package-checkins@software.sandia.gov CVS commit emails. “Finger on the pulse” list. package-developers@software.sandia.gov Mailing list for developers. package-users@software.sandia.gov Issues for package users. package-announce@software.sandia.gov Releases and other announcements specific to the package. New_package (optional): Customizable boilerplate for Autoconf/Automake/Doxygen/Python/Thyra/Epetra/TestHarness/Website

Sample Package Maturation Process: 

Sample Package Maturation Process Startup Steps Maturation Steps

Maturation Jumpstart: NewPackage: 

Maturation Jumpstart: NewPackage NewPackage provides jump start to develop/integrate a new package NewPackage is a “Hello World” program and website: Simple but it does work with autotools. Compiles and builds. NewPackage directory contains: Commonly used directory structure: src, test, doc, example, config. Working Autoconf/Automake files. Documentation templates (doxygen). Working regression test setup. Working Python and Thyra adaptors. Substantially cuts down on: Time to integrate new package. Variation in package integration details. Development of website. NOTE: NewPackage can be use independent from Trilinos

What Trilinos is not: 

What Trilinos is not Trilinos is not a single monolithic piece of software. Each package: Can be built independent of Trilinos. Has its own self-contained CVS structure. Has its own Bugzilla product and mail lists. Development team is free to make its own decisions about algorithms, coding style, release contents, testing process, etc. Trilinos top layer is not a large amount of source code: Trilinos repository contains 452,187 source lines of code (SLOC). Sum of the packages SLOC counts : 445,937. Trilinos top layer SLOC count: 6, 250 (1.4%). Trilinos is not “indivisible”: You don’t need all of Trilinos to get things done. Any collection of packages can be combined and distributed. Current public release contains only 18 of the nearly 30 Trilinos packages.

Insight from HistoryA Philosophy for Future Directions: 

Insight from History A Philosophy for Future Directions In the early 1800’s U.S. had many new territories. Question: How to incorporate into U.S.? Colonies? No. Expand boundaries of existing states? No. Create process for self-governing regions. Yes. Theme: Local control drawing on national resources. Trilinos package architecture has some similarities: Asynchronous maturation. Packages decide degree of interoperations, use of Trilinos facilities. Strength of each: Scalable growth with local control.

Solver Collaboration: The Big Picture: 

Solver Collaboration: The Big Picture

Solver Software Components and Interfaces: 

ANA Linear Operator Interface Solver Software Components and Interfaces 2) LAL : Linear Algebra Library (e.g. vectors, sparse matrices, sparse factorizations, preconditioners) ANA APP ANA/APP Interface ANA Vector Interface 1) ANA : Abstract Numerical Algorithm (e.g. linear solvers, eigen solvers, nonlinear solvers, stability analysis, uncertainty quantification, transient solvers, optimization etc.) 3) APP : Application (the model: physics, discretization method etc.) Example Trilinos Packages: Belos (linear solvers) Anasazi (eigen solvers) NOX (nonlinear equations) Rhythmos (ODEs,DAEs) MOOCHO (Optimization) … Example Trilinos Packages: Epetra/Tpetra (Mat,Vec) Ifpack, AztecOO, ML (Preconditioners) Meros (Preconditioners) Pliris (Interface to direct solvers) Amesos (Direct solvers) Komplex (Complex/Real forms) … Types of Software Components Thyra ANA Interfaces to Linear Algebra FEI/Thyra APP to LAL Interfaces Custom/Thyra LAL to LAL Interfaces Thyra::Nonlin Examples: SIERRA NEVADA Xyce Sundance … LAL Matrix Preconditioner Vector

LAL Foundation: Petra: 

LAL Foundation: Petra Petra provides a “common language” for distributed linear algebra objects (operator, matrix, vector) Petra provides distributed matrix and vector services. Has 3 implementations under development.

Petra Object Model: 

Perform redistribution of distributed objects: Parallel permutations. “Ghosting” of values for local computations. Collection of partial results from remote processors. Petra Object Model Abstract Interface to Parallel Machine Shameless mimic of MPI interface. Keeps MPI dependence to a single class (through all of Trilinos!). Allow trivial serial implementation. Opens door to novel parallel libraries (shmem, UPC, etc…) Abstract Interface for Sparse All-to-All Communication Supports construction of pre-recorded “plan” for data-driven communications. Examples: Supports gathering/scatter of off-processor x/y values when computing y = Ax. Gathering overlap rows for Overlapping Schwarz. Redistribution of matrices, vectors, etc… Describes layout of distributed objects: Vectors: Number of vector entries on each processor and global ID Matrices/graphs: Rows/Columns managed by a processor. Called “Maps” in Epetra. Dense Distributed Vector and Matrices: Simple local data structure. BLAS-able, LAPACK-able. Ghostable, restributable. RTOp-able. Base Class for All Distributed Objects: Performs all communication. Requires Check, Pack, Unpack methods from derived class. Graph class for structure-only computations: Reusable matrix structure. Pattern-based preconditioners. Pattern-based load balancing tools. Basic sparse matrix class: Flexible construction process. Arbitrary entry placement on parallel machine.

Petra Implementations: 

Petra Implementations Three version under development: Epetra (Essential Petra): Current production version. Restricted to real, double precision arithmetic. Uses stable core subset of C++ (circa 2000). Interfaces accessible to C and Fortran users. Tpetra (Templated Petra): Next generation C++ version. Templated scalar and ordinal fields. Uses namespaces, and STL: Improved usability/efficiency. Jpetra (Java Petra): Pure Java. Portable to any JVM. Interfaces to Java versions of MPI, LAPACK and BLAS via interfaces.

Data Model Wrap-Up: 

Data Model Wrap-Up Our target apps require flexible data model. Petra Object Model (POM) supports: Arbitrary placement of vector, graph, matrix entries on parallel machine. Arbitrary redistribution, ghosting and collection of distributed data. This flexibility is needed by LALs: Algebraic and multi-level preconditioners. Concrete distributed matrix kernels. Direct methods. POM is complex: Non-LALs (ANAs) do not rely on it.

Slide31: 

Full “Vertical” Solver Coverage Trilinos Packages

Scalable Preconditioners: 

Scalable Preconditioners

Preconditioner Efforts: 

Preconditioner Efforts ML: Many applications can use if matrix is SPD, or close. Continuing efforts in custom operators. CLAPS: Clark’s Linear Algebra PackageS. Segregated preconditioners: Meros. Mortar methods: New work. IFPACK: DD type algebraic preconditioners. Numerous other efforts focused on specific applications. Theme: Block preconditioners. Bottom line: Scalable, robust preconditioners are essential.

Software Quality: 

Software Quality

SQA/SQE: 

SQA/SQE Software Quality Assurance/Engineering is important. No longer sufficient to say “We do a good job”. Trilinos Developer Guide Part II must be remapped to current ASC SQE 30 (instead of 47) practices.

2005/2006 Future Plans: 

2005/2006 Future Plans Trilinos Package Architecture: Continue refinement of new_package. Status: Must revisit. Explicitly define Trilinos compatibility. Status: Done. Resolve the abstract interface issue. Status: Done. Software Quality: Expand use and ease-of-use of test harness. Status: Done. Identify metrics and automate capture and display. Status: Got dropped. Establish a life-cycle model (hybrid agile/unified process?). Status: Good progress. Customize the ASC SQP to our environment. Status: Must restart. Packages: Foster new package development. Status: Package count still growing Manage the growth. Issue: complexity of package coupling. Status: Some new concerns. Harden our mature packages. Status: Some progress, much more needed. Transition to post-delivery maintenance: Status: Still unresolved. Organizational issue: Tough to solve.

FY06/07 Themes StatusFramework: 

FY06/07 Themes Status Framework Take Steps toward dynamic package addition. Needed to enhance external interoperability. Trilinos-compatibility definition or something similar. Status: Done. Define SW Lifecycle(s) and begin formalized efforts. Need something for external audits that is still useful to us. Agile vs. UP vs. hybrid. Status: Good progress. Make Trilinos more accessible to new users. We have lots of introductory material. Need to organize it for new users. Status: Good progress.

FY06/07 Themes StatusAlgorithms and Development: 

FY06/07 Themes Status Algorithms and Development Multi-level preconditioners. Curl-curl. Mortar methods. DD algebraic preconditioners. Xyce-focused. Parallel KLU. Complex linear solvers. Optimization. Transient solvers. Eigensolvers. Maturation of Thyra and Python adapters. Matrix loading interface in Thyra. Public release of Belos, CLAPS, Galeri, Komplex, Meros, Rythmos, Tpetra, …

FY07/08 Themes: Framework: 

FY07/08 Themes: Framework Trilinos Level II Milestone. Demonstrate Use of Full Vertical capabilities in Charon. Joint licensing and copyright of development with other organizations. Other DOE Labs, international orgs, private companies. Package Autonomy (reacting to rapid growth): 18  27  ?? Guarding against incidental coupling. Revisiting the location of “skins”. Stratimikos: Uniform access to many packages. Access from Fortran. DOE Science Users, DOD Users. Stratimikos focus. Split of User vs Developer tools. http://trilinos.sandia.gov http://software.sandia.gov

FY07/08 ThemesAlgorithms: 

FY07/08 Themes Algorithms Our role in V&V. Tim Trucano’s talk. Much of the same work, with different drivers and goals. Full Vertical solution capabilities. Optimization (PDE type). Charon. A push more than pull. (A concern). Ever better forward solves. Critical enabler. ML, Ifpack, Paraklete, Belos. Specialized Preconditioners: Aggregation of baseline tools.

Tools on software.sandia.gov: 

Tools on software.sandia.gov CVS: Source management. Bugzilla: Bug/feature tracking. Mailman: Mail list management. Bonsai: Interface to CVS/Bugzilla. Webserver: Webpages accessible from anywhere. Autotools: Autoconf/automake facilities. Wiki: Online logging tool.

Trilinos Availability/Information: 

Trilinos Availability/Information Trilinos and related packages are available via LGPL. Current release (7.0) is “click release”. Unlimited availability. Trilinos Release 8.0: April 2007, 8.1: September 2007. Trilinos Awards: 2004 R&D 100 Award. SC2004 HPC Software Challenge Award. Sandia Team Employee Recognition Award. Lockheed-Martin Nova Award Nominee. More information: http://software.sandia.gov http://software.sandia.gov/trilinos Additional documentation at my website: http://www.cs.sandia.gov/~mheroux.

Conclusions: 

Conclusions Trilinos services to developers and users: Infrastructure, Interfaces, Implementations. Simplifies installation, support for users of total collection. Epetra, Teuchos & Thyra promote common APIs across all other Trilinos packages. Each package can be built, used independently (with only explicit dependencies), and exists as independent project. Primary goals: Rapid development and installation of robust numerical solvers. High-quality production software for the critical path. More information: http://software.sandia.gov http://software.sandia.gov/trilinos Additional documentation at my website: http://www.cs.sandia.gov/~mheroux.