56901

Case Studies in Prediction and Prevention of Failures : 

Case Studies in Prediction and Prevention of Failures Chandra X-ray Observatory Propulsion Subsystem Anomalies Sabina Bucher

Purpose: 

Purpose Share Chandra program Propulsion anomaly experience Trace two anomalies through Initial Anomaly detection Analysis techniques Mitigation strategies Implementation of mitigating actions Review Best Practices and Lessons Learned

Background: 

Background Chandra is a Space based telescope Maneuvers several times a day One side always faces sun Passive thermal controls degrading Changing sun angles cause large temperature variations on some components

Case 1: Reduction in Thrust: 

Case 1: Reduction in Thrust The Momentum Unloading Propulsion Subsystem (MUPS) thrusters are used to unload accumulated angular momentum An operational unload took 60% longer to complete than expected Attributed to reduced thrust from one of the sun-side thrusters Pulse-by-pulse performance of individual thrusters assessed Contributing factors identified Scheduling of operational momentum unloads modified Scheduling modifications have prevented reoccurrence of the anomaly

Momentum Management: 

Momentum Management Reaction Wheels provide maneuver capability and pointing control for Chandra Reaction wheels store angular momentum accumulated from solar wind and gravity gradient effects Propulsion subsystem unloads accumulated angular momentum All unloads planned and executed by stored command sequence Stored angular momentum tracked and propagated carefully An unload is planned whenever the stored momentum is predicted to exceed the operation threshold

Calibration of Thruster Performance: 

Calibration of Thruster Performance Derivation Choose 12 unloads All axes well represented All thrusters well represented Perform least square fit of on-time to delta momentum telemetry Uses Provide accurate ground based modeling of momentum unloads Invaluable in monitoring and measuring thruster performance

Observed Reduction in Thrust: 

Observed Reduction in Thrust Nominal Unload Nominal unloads achieve near linear transition from one state to the next Anomaly best modeled by cutting the thrust provided by one of the thrusters to 55% of its nominal value 360 seconds into the unload

Pulse-by-Pulse Performance: 

Pulse-by-Pulse Performance

Identifying Contributing Factors: 

Identifying Contributing Factors Thruster efficiency calculation used to process all momentum unloads Six exhibited the anomaly signature Three showed indications of the anomaly Anomalous unloads searched for common traits Temperature and unload duration identified Modeling performed at the factory supported these observations

Defining and Implementing Scheduling Constraint: 

Defining and Implementing Scheduling Constraint New Constraint: do not perform momentum unloads with duration over 600 s or starting temperatures over 120º F Limiting Durations: Choose the unload target to shorten the unload duration Use deadman to cut off any unload that runs long Limiting Temperatures: Required a model that could predict thruster temperature Developed an empirical model Incorporated model into the Mission Planning suite of tools

Monitoring Thruster Performance: 

Monitoring Thruster Performance Every unload since anomaly detection has been checked with the thruster efficiency calculation No additional occurrences have been found Implementing such a technique on other programs Several weeks of up-front effort Now takes less than 30 s to run Useful in trending thruster performance Can highlight performance changes indicative of impending failure

Case 2: Cold Sun-Side Feedlines: 

Case 2: Cold Sun-Side Feedlines “Caution low” limit violations on two sun-side propulsion line thermistors Brief Infrequent Always on attitudes that put the sun on the tail of the vehicle Trending data dominated by solar heating Heater cycle turn on temperatures isolated Revealed steady, mission long cooling trend Feedline temperature profiles characterized with respect to time and attitude Limited duration of dwells at orientations requiring the heaters Cooling trend successfully halted

Propulsion Thermal Protection: 

Propulsion Thermal Protection Propulsion lines on Chandra are spiral wrapped in multi- layer insulation (MLI), heaters, and aluminum tape Heaters controlled by bi-metallic thermostats Cannot be commanded on Cannot be re-programmed Every set of thermostats controls a circuit of heaters Temperature telemetry provided by thermistors located at various points along the lines The propulsion subsystem wraps around the front of the spacecraft bus Attitudes that put the sun at the tail of the vehicle (tail-sun) put the propulsion components into shadow

Cold Temperatures on Sun Side Feedlines: 

Cold Temperatures on Sun Side Feedlines Brief and Infrequent Always at tail-sun attitudes Heaters turning on late Once on, heaters functioning well To keep other units cool, time spent at tail-sun attitudes increasing Thermistors B and C well removed from the thermostats On orbit changes have caused the thermostats to stay warm longer than remote sections of line once in shadow Allows portions of the propulsion lines to be exposed to cold temperatures Attitude Dependence Limit Violations on Thermistors B and C Sun

Isolating Heater Cycles: 

Isolating Heater Cycles Mission long data set split into attitude by attitude segments Temperatures of each segment analyzed independently Calculated rate of temperature change for each attitude Makes heater turn-on obvious Used to collect statistics on heater cycles Revealed mission long cooling trend Attitude-by-attitude telemetry showed Thermistors A and B behaved differently

Temperature Behavior: 

Temperature Behavior

Defining and Implementing a Scheduling Constraint: 

Defining and Implementing a Scheduling Constraint Thermistor A constraint: do not schedule attitudes past 170 deg sun-pitch Small operational impact Easily implemented Thermistor B constraint: do not schedule attitudes past 150 deg sun-pitch Eliminates attitudes used to cool other spacecraft components Makes some time-constrained science observations impossible Constraint needed more balance Keep propulsion lines safely above freezing, Do no eliminate tail-sun attitudes Thermal model considered Data too sparse Too many factors contributing to the final temperature of the lines Required high degree of accuracy

Determining Maximal Cooling Rates: 

Determining Maximal Cooling Rates Cooling envelope establishes the maximum cooling rate from one temperature to another Maximal cooling rates used to set “do not exceed” limits on time at cold attitudes

Implementing Scheduling Constraint: 

Implementing Scheduling Constraint Preheat lines before maneuvering to a cold attitude Minimum preheating times set with plots of temperature vs. elapsed time at attitude Implementation without software very labor intensive Additions to existing software tools made implementation manageable New Constraint New Tool Plot sun angle vs. time Show transitions into and out of propulsion line regions Issue red warning if: 1) pre-heating requirement not met 2) maximum duration exceeded

Conclusions: 

Conclusions Successful mitigation of two Chandra Propulsion Anomalies Initial indications subtle In-depth analysis revealed distinct performance changes Adaptive Mission Scheduling process allowed successful mitigation Incorporate in-depth analysis methods into day-to-day operations Software and Hardware advancements making this increasingly feasible Identify problems before they become failures Mission Scheduling can evolve gracefully as the vehicle ages Provide the scheduler with all of the information that goes into scheduling Allow the scheduler to specify how requests are scheduled Use optimization routines aid, but not replace, the scheduler Chandra’s remarkable safety and efficiency record contributed to by: An environment that fosters in-depth analysis An adaptive approach to mission scheduling