8 management rek RyadElKhatib

From source code management to binaries production for Arpege/Aladin/Arome: 

From source code management to binaries production for Arpege/Aladin/Arome R. El Khatib Météo-France – CNRM/GMAP November, 2005 General considerations Code management at GMAP The « gmkpack » tool

Source code management issues: 

Source code management issues Huge software ( 500 000 lines) => Organization Numerous developers ( 75 to 100 persons) => Phasing Perpetual changes (innovations, debug, …) => Historization Multi-purposes (operations, research …) => Versioning

What does a source code management tool offer ?: 

What does a source code management tool offer ? Continung principles : Code partitioning in a database Stamped modifications More or less automatic merging operations Continuing problem : The fear (?) of developers to use a complex (?) tool The need to learn how to use the tool An investment for the code maintenance

Arising problems with code evolutions: 

Arising problems with code evolutions Externalizations, modularizations :  : More flexibility and longer life : Manual links between object libraries : Interdependencies of libraries through inclusion files, modules and duplicated subroutines Programming languages :  : more structured and more robust : Dependencies of modules => hierarchical compilation Non-standard (odb,bl) compilers arising Paradoxally … It is more and more difficult to make binaries !!!

From compilation to binaries: 

From compilation to binaries In the old time, before Fortran 90 : …There were nothing special ! (too easy ?) Since F90, odb90.x and « duplicated stuff » : Makefiles : can only partly solve the problem Dedicated scripts : become mandatory Many initiatives to develop Perl-based tools

Code management at GMAP: 

Code management at GMAP Source codes inside a database (ClearCase) : to view or modify Separate html source code browser GCO staff : « Management of Operational Cycles » « Packs » : copy of source code, object code, libraries and executables on various platforms Developers : personnal « packs » based upon GCO’s main « packs »

Structure of « packs »: 

Structure of « packs » Arborescent structures containing : bin/ : binaries lib/ : libraries src/ : source and object code sys/ : specific tools (precompilers) Below src/ : A « ClearCase-view-like » structure of branches (ex : main/ bf/) Below each branch : all the « vobs » (projects) (ex : arp/ ald/ tal/ tfl/ xrd/ odb/ etc …) Below each vob : a « ClearCase-vob-like » structure of code (ex : setup/ control/ etc …)

Management of « packs »: 

Management of « packs » Strict naming conventions for the background packs Example : cy24t1_export-bf.01.L9912.x.pack/ Strict naming conventions of the files (cf Norms …) Strict definitions of object libraries : Ordering (ex : ald before arp) Content (ex : bf_[n] always included in bf_[n+1] Leading arpege release Branch name Version number Compiler release Compilation options

Current operations on source code: 

Current operations on source code Local code modification Local pack elaboration Local pack usage Use ClearCase to make a « branch » Procedure « cc_export » to create a user pack and populate it from the ClearCase branch Home-made Procedure « gmkpack » to work from compilation to binary elaboration

gmkpack: 

gmkpack To compile, make object libraries and binaries in a « pack » environment Specifications : Universal (serving anybody working on arpege/aladin/odb/arome/…) Secure (specially : in libraries/compilation ordering) Flexible (choice of compiling options, libraries …) Users friendly (simple interface, automatic analysis of what should be recompiled/rebuilt) Efficient (fast commands, fully distributed compilation) Portable (any platform, as an independent tool)

Slide11: 

Gmkpack : genesis (1) Wrapper of a « universal » makefile which did not succeded : « mkpack » Home-made procedure in Perl : efficient but not suitable for developpers : « gmak » => a long way to achieve the first user-friendly wrapper to gmak : Gmkpack.4.4 = « version 1 » (2001)

Slide12: 

Gmkpack : genesis (2) Gmkpack.5.1 = « Version 2 » : designed to be platform-independent (2003) Ability to build reference environments (full cycle) Ability to run on other system than the VPP Gmkpack.6.1 = « version 2 ½ » : Stable framework for future evolutions

Slide13: 

Gmkpack : how does it look like ? Creation of a reference « pack » by an administrator : % gmkpack –a –r 29T2 –p aladin src/local/ lib/ bin/ sys/ ics_aladin* Developpers’s modset upon the reference « pack » : % gmkpack –r 29T2 –b my_mods –p aladin src/local/ src/29T2 lib/ bin/ ics_aladin*

Slide14: 

Gmkpack : recent evolutions Support for Meso-NH source code : New vobs « mpa/ » and « mse/ » Files extension « .mnh »  .f90, implicit-kinded variables Norms checker report in dayfile Upgrade : A bit more secure Re-build of compilation scripts still needed 1 new environment variable (GMKTMP) No incremental update

Slide15: 

Gmkpack : future evolutions Finalize the portability (LINUX, cross-compilation platforms) Best efficiency needs to re-write all in Perl Improve the parallelism (source code growing !) « Externalize » the source code specificities Enable alternative architecture files for a given platform Revisit the libraries managements (do as the source code)

Slide16: 

Gmkpack : timetable for next releases Timetable not planned ! Content not planned ! Intermediate releases will come out anyhow : to fix bugs for miscellaneous improvment A major upgrade can be expected at the time Météo-France will change its supercomputer (in coordination with « GCO »)

Conclusion/perspectives: 

Conclusion/perspectives Source code manager : Choice of the more appropriate application Source code browser : to be updated/re-written following the evolution of the code Packs structure : needs enhancements gmkpack : stable but still too young