OPS-G ForumXMM-Newton MOC and SOC migration to SCOS-2000: OPS-G Forum XMM-Newton MOC and SOC migration to SCOS-2000 G. Kerr, J.D. Ponz 23rd September 2005


Contents: Contents Purpose of the project Why migrate ? XMM ground segment Original XMCS and XSCS in numbers Sub-systems to be migrated Migration approach Schedule and budget Migration at the MOC Migration at the SOC Project status Critical steps: Extensive tests Adapt operational tools Lessons learnt


Purpose of the project: Purpose of the project Migrate the XMM-Newton ground control system of both MOC and SOC from the old SCOS-1 based system to SCOS-2000 while Maintaining the functional specifications Maintaining / improving system performance Maintaining the systems interfaces (except MOC-SOC link)


Why migrate ?: Why migrate ? Mission extension approved till 2008 and could be extended to 2012 … Reduction of long-term maintenance cost Long-term improved maintainability Reduce risks in hardware and software Common infrastructure at MOC and SOC Common inter-project infrastructure in shared areas at ESOC (e.g.: MCR) Follow ESA investment policy: SCOS-2000/EGOS


XMM ground segment: XMM ground segment observations POS EPOS timeline NCTRS TCS MPS FDS SGS TMS PMS QLA ODS AMS SAS PPS External SOC MOC Ground Stations TC TC, TM TM TM + Aux PHS proposals ODF, SDF Observer observation results XMCS XSCS AMS: Archive Managent; FDS: Flight Dynamics; MPS: Mission Planning; NCTRS: Network Control; ODS: Observation Data; PHS: Proposal Handling; PMS: Payload Monitoring; PPS: Pipeline Product; QLA: Quick-look Analysis; SAS: Science Analysis; SGS: Sequence Generator; TCS: Telecommanding; TMS: Telemetry


The Original XMCS and XSCS in Numbers: The Original XMCS and XSCS in Numbers Biggest single contract software development for ESOC Total: 17 M€ XMCS: 2.6 M€ (SR-AD-DD) XSCS: 11 M€ (SR-AD-DD) XMCS+XSCS : 3.4 M€ (OM in 2000) Up to 70 developers in DD 286k lines of code developed on top of SCOS-1 (~250 LoC) High efficiency: 13 LoC/day (10 LoC/day is standard) GC (former OPS-GD) provided technical and contract management from 1993 to end Dec 2000 4 staff (170 mm or ca. 3 M€) ==> 17 % contract cost 0 2 4 6 8 10 12 M € SR AD DD OM Phases Griesheim Griesheim 0 10 20 30 40 50 60 70 # D e v S t a f f SR AD DD OM Phases London(55) Rome(15) ESOC(4) Vilspa(12)


Sub-systems to be migrated: Sub-systems to be migrated All MOC Subsystems SOC subsystems using SCOS-1 infrastructure: Payload monitoring subsystem (PMS) Observation data subsystem (ODS) Quick-look analysis subsystem (QLA) Database subsystem (DBS) Trend Analysis subsystem (Speval->TDRS +TDAS) Criteria: Maintain/improve operational functions Maintain interfaces (ICDs) [except MOC-SOC link] Final test: byte-to-byte comparison of output products (ODFs/SDFs)


Migration approach: Migration approach Make use of basic SCOS-2000 functionality Use generic TM packetiser Use generic TC subsystem (modified for XMM) Use generic OBSM subsystem (modified for XMM) Replace SPEVAL by TDRS Share XMCS – XSCS subsystems Achieve maximum synergies Minimize migrated code Port mission specific components Replace SCOS-1 interfaces by SCOS-2000 where possible Automatic code conversion where possible (PMS) Adapt auxiliary tools to the migrated system Trend analysis, FOPTool, Timeline tool, …


Schedule: Schedule


Budget: Budget MOC Awarded after competition to LogicaCMG KO in Oct 2002 FFP 750k€ + 23k€ (migration to S2k 3.1) FUP 23k€ (flexibility + support) SOC Awarded after competition to LogicaCMG KO in Mar 2003 FFP 560k€ FUP 97k€ (flexibility + support) + 65k€ (transferred from ERS-2000)


Migration at the MOC (1): Migration at the MOC (1) Original planned deliveries were: R1 : DBS subsystem R2: MPS, OBSM, NIS, XFTS R3: Bug fixing R4: Final MOC delivery (full functionality) System to be based initially on S2K-R2.3e, and upgraded to S2K-R3.0e, some months after KO. Actual MOC deliveries included R5, R6, and R7. Currently on R8.8, combined MOC+SOC delivery)


Migration at the MOC (2): Migration at the MOC (2) Decision to keep Derived Parameters in Fortran with C++ wrappers Built on Logica ERS database conversion experience – very useful basis Replacement for SPEVAL essential (use of RAPID) Unfortunate start with S2K3.0 – many problems, followed by infrastructure decision not to maintain 3.0 S2K problems forced us to port to 3.1 at short notice Fortunately SOC conversion started months after MOC, allowed cost sharing and synergy in 3.1 port Flexibility WP excellent idea… Decision to switch both MOC+SOC together…


Migration at the SOC (1): Migration at the SOC (1) Deliveries were R1: Combined MOC-SOC database editor R2: PMS core functions, excluding instrument processors R3: Instrument processors, QLA and ODS R4: Full system Base system: SCOS-2000 r3.1 Actual SOC deliveries included also R5. Currently on R8.6 combined MOC+SOC delivery.


Migration at the SOC (2): Migration at the SOC (2) The port to 3.1 at the MOC delayed R2, R3 No impact on the overall schedule DBS subsystem required additional releases Conversion of Derived parameters, Consistency checker, … System integration and testing: Initially at ESOC (R1, R2) then at ESAC Concurrent computer procurement at both sites Extensive validation of mission products Migration of network infrastructure during the project


Project status: Project status In full operations since 14 June 2005 S2K based system Communication lines Output data products sent to SSC: pipeline process Project objectives are achieved Fulfilled on time and within budget (25 % savings !) Full customer satisfaction [Ops + Sci + … scientists] Fully transparent to SSC (pipeline production)


Critical step: Extensive tests: Critical step: Extensive tests Standard system tests in validation chain. Specific live tests: LT-1: 24-28 Nov 2004. SCOS-1 commanding. S2k commanding disabled. All instrument modes to exercise instrument processors. LT-2: 25-27 Jan 2005. First test commanding with SCOS-2000. Exercise commanding for all instrument modes. Successful but MOC-SOC link problems LT-3: 2-4 March 2005. Second test commanding with SCOS-2000. Successful but AOCS anomaly [not due to SCOS-2000] Parallel ops with SCOS-1 commanding – 5 April 2005 Switch to SCOS-2000 commanding – 14 June 2005 Real-time regression tests with the simulator using predefined timeline Systematic regression tests using rev. 909 (LT-1) Continuous comparison of ODF products from both chains


Adapt operational tools: Adapt operational tools Trend analysis tool from SPEVAL to TDRS Generic software, easy to use in other missions Implemented in IDL - common SOC environment The system includes C++ interface library TDRS request (batch or via GUI) Trend analysis algorithms for all instruments in IDL Graphical and numerical reporting and web publishing Used to monitor instrument performance Developed as collaboration OPS-G (J. Barreto) and OPS-O (J. Fauste)


Lessons learnt: project specifics: Lessons learnt: project specifics A migration project is a very special enterprise: Because users (and tools) are adapted to the “old” system Therefore, it is essential to identify the advantages of the new system to compensate the effort But if the system works, it is not perceived as a merit (it was already working …) A good point: it is possible to validate the migrated system in a more rigorous form


Lessons learnt: why it was a success ? : Lessons learnt: why it was a success ? Successful: on time, on budget with satisfied customers Porting made possible because it was planned in advance (1999) with funding approved in XMM CaC Involvement of all project actors to use their experience (OPS-O, D/Sci) Cooperation and motivation of SOM and FCT Positive relationship with contractor. Having only 1 contractor –LogicaCMG- for both MOC and SOC highly valuable Collaboration: Shared problems, involvement in decision process MOC-SOC coordination was essential Extensive testing and product verification Strict configuration control Consistent distribution of information


Lessons learnt: complex project: Lessons learnt: complex project GS complexity from OPS-G perspective MOC: Critical functions, but Only one customer [OPS-O] Limited number of subsystems SOC: Three customers [OPS-O + D/Sci + SSC] Open ended number of subsystems Distributed MOC-SOC => Coordination Common SW releases Agreed timelines


Lessons learnt: But … : Lessons learnt: But … Only one migration project at a time (three in our case: SCOS-2000, network infrastructure and AMS). Significantly underestimated number of S2K problems – often had to be fixed by contractor in advance of infrastructure fixes (otherwise system testing blocked, disruptions in time plans). Underestimated effort involved in integrating major S2K patch deliveries, with changes in hundreds of source files (3 deliveries so far, 4th on the way – not budgeted!!). DBS subsystem should be better integrated in the system infrastructure.


Still To Be Done: Still To Be Done Migration from TCP/IP to X25 – now started – could potentially be problematic because of implicit timing assumptions in DB (~ many weeks of testing) Migration to TMTCS and SLE – next spring? (requires phase 2 of TMTCS for SLE AD mode) SOC LAN reconfiguration Conversion of 800GB of existing SPEVAL data to S2K format Integration of S2K M02/M03 underway (end OCT..) Integration of M04 not yet decided (10 out of 170 SPRs might be interesting…) Integration of future S2K releases (R4.0…) not budgeted at this time


Conclusion: Conclusion “Our Operators at ESAC are very happy about the porting and they would not like to go back to SCOS-1 !” (M. Casale, OPS-OFX/ESAC, leader of the instrument operations team dixit)