Confusing Data and Assumptions about Metallurgy

On September 1, 2016 a Space Ex Falcon 9 exploded on the pad at Kennedy Space Flight Center. The Falcon was in the process of loading propellants when this explosion occurred. Preliminary reports suggest that a large breach occurred in a helium tank in the upper stage which after some .9 seconds caused the rupture of the LOX tank it was contained within. The resultant fire destroyed much of the Falcon 9 and severely damaged the pad itself.

There is some careful wordsmithing going on at the moment to suggest that this accident has no relationship to the earlier Falcon 9 that exploded in flight but the truth of the matter is that the ultimate destruction of both vehicles was caused by the failure of a helium tank inside the upper stage LOX tank. Space Ex, who had earlier determined that the first loss was caused by the failure of a mounting strut that holds the tank in place during flight, immediately suggested that there is no correlation between the two failures as this strut is not under dynamic load during propellant loading. While this is true, it skips over the fact that only Space Ex believes that it fully understands what happened to the first flight. An independent NASA investigation into the same incident suggests that while the strut issue was a problem, there are several other possibilities that could have caused the same incident. In other words, while everyone agrees that a rapid overpressurization of the LOX tank caused the incident, everyone does NOT agree about what caused the failure of the helium tank that led to this overpressurization.

The first Falcon 9 that exploded in mid-flight was most definitely experiencing dynamic loads that are not present during propellant loading on the pad. Therefore, it is extremely unlikely that a strut failure occurred to cause this explosion. However, it is worth backing up a little bit here to explain how Space Ex decided that the strut failure was actually what caused the first accident.

Early data from the first incident presented some seemingly conflicting and contradictory data. Telemetry system data suggested that there was a very brief drop in Helium pressure immediately previous to the explosion. This would make sense if a tank experienced a sudden leak or breach of some kind but there was also data that suggested that the pressure immediately returned to normal before the actual explosion. Engineers from Space Ex and NASA were confused by this information to say the least.

Early on, Space Ex was concerned about the bouyancy effect of the Helium tanks within the LOX tank. Bouyancy in LOX is little different than bouyancy in water and most everyone understands that holding a balloon underwater is problematic. The same thing occurs in a COPV pressurized with helium in a LOX tank. As the G forces increase during launch the bouyancy increases. In other words, the upward pressure on the struts that hold the helium tanks in place increase as the rocket ascends rapidly. Due to the timing of the incident on the Falcon 9 it seemed that this problem occurred simultaneous with a very high G loading on the flight.

The second clue that led investigators to look at these struts was some acoustic data from microphones/accelerometers on the vehicle. The data from these intruments is taken at a high rate of speed that is inherently necessary to gather vibration data for analysis. As I have discussed in here before (see Update Rates) digital data systems take snapshots of pressure, vibration, and temperatures. These snapshots are taken at varying rates dependent on the type of data you are trying to collect. These snapshots are then arranged on a plot and a line is drawn between the points on the chart to create a graph displaying this information vs. time.

One of the problems with digital data is that is can easily be used to draw graphs that do not resemble actual events. For instance, if you take digital data on a repeating sine wave that operates once a second you can accurately represent this sine wave if you take at least ten snapshots per second. However, if you take only two snapshots in this time period you will wind up with a graph that doesn't even resemble a sine wave. It is accurate data at that point but it completely misrepresents a sine wave. In other words, the snapshots are accurate but the resultant graph is bogus.

There are standard formulas for deciding data rates for all manner of instrumentation which I won't go into here, but it is also dependent on both the instrument you are using to gather the data and the speed of the event you are trying to capture. In the case of the Falcon 9 that exploded in flight the data gathered from the Helium pressurization system was probably rather slow in terms of trying to capture the event that actually happened. I don't know this for a certainty as I have not personally seen the data but a pressure trandsducer that is being used to monitor tank pressure there is typically no need to monitor it at a high rate of speed as the pressure is not expected to change extremely rapidly. If you knew you might want to use it to decide exactly how and when something explodes you would run it at an extremely high rate of speed but that is not what this system was designed and built to do.

Ideally, one would set all such systems up for such an eventuality but running at extremely high rates of speed on numerous channels costs money and if you aren't convinced you will ever need this high speed data you simply don't design systems to accomodate it. When we are purposefully taking COPV's to failure we would typically run our data collection pressure channels at 50,000 Hz. In other words, we would take a snapshot 50,000 times a second so that we could see exactly what the pressure was when the tank ruptured. Gathering 50,000 Hz data is not that difficult with today's systems but storing it and being able to analyze it later can be problematic and expensive.

I suspect from the information I have seen released that the pressure system Space Ex was using on the helium tanks was on the order of 10 hz. Again... I don't know this for certain but it would make sense economically and technically as they were not expecting to see rapid pressure changes in this system to begin with. It would also explain the delay between the drop in pressure and the overpressurization or loss of the vehicle. Even if they were running at 100 hz it is still not fast enough to have a lot of data as to what had actually happened in the helium system.

Space Ex was also running accelerometer/microphone data systems at several locations on the vehicle to monitor vibration during the flight. These types of instruments operate at much higher frequencies inherently because they are looking for vibration signals in the thousands of cycles per second. Using this high frequency information they look for vibrational problems that might create positive feedback loops of resonant frequencies that could damage or destroy the vehicle. This is another problem with space flight that is always a concern. You don't want to set up positive feedback loops that destroy your vehicle. Instrumentation looking for these loops is monitored and systems are throttled specifically to avoid these issues.

Using this high speed data, Space Ex determined that there were two significant incidents that occurred at different times. The first was a significant "sound" or detectable vibration and .9 seconds later the vehicle exploded. By triangulating the various signal locations Space Ex determined that the first "sound" came from the area where the helium tanks were located. This led back to their original concern about the bouyancy of the helium tanks in LOX and how it was affected by the G loading during ascent. If the original "sound" was a strut breaking, the rapid rise of the helium tank in the LOX tank could have followed. The resultant collision with a wall of the LOX tank or the top of the tank would have precipitated the instantaneous failure of the COPV, releasing 5500 PSIG of Helium into the LOX tank and overpressurizing it immediately.

Space Ex began testing mounting struts that they had in stock to see if some of them might fail at lower torque ratings than their specification. What they found was that several of them did fail at much lower ratings than their specification. Meallurgy studies found consistency problems at the granular level in these struts. Steel struts, just like steel bolts are rated for shear strength and manufactured accordingly. The second part of this problem has to do with the fact that they were being used to mount helium tanks inside a LOX tank. If a steel strut has minor inconsistencies in the granular structure but it is highly overrated for shear pressure this is not a problem. Space Ex soon came out and said that some of these struts failed at 5 times lower pressures than they were rated for. They also refused to release the name of the manufacturer of these struts but stated that from this point forward they would individually test each strut before use.

All of this sounds reasonable except for there is no mention of the more important fact that typically carbon steel struts are NEVER used in cryogenic applications. Exposing stainless steel to -297 degrees temperature changes its shear rating dramatically. Minor granular inconsistencies become major catostrophic failures under these conditions, which is why you do not use any type of steel struts in such conditions without comprehensive metallurgical pedigrees. Space Ex has been in a constant running battle with its NASA oversight groups from the beginning of its existance because of its unwillingness to do due diligence on configuration control issues such as this one. When this Falcon 9 exploded in mid-flight there were numerous parts flying on it that Space Ex could not identify as to origin or pedigree. This was not limited to struts, nuts, and bolts but went as far as valves, regulators, and all manner of complex components. In other words, they completely lost configuration control in their haste to launch vehicles on many of the systems on their vehicles. This is a much worse problem than a few struts that failed during testing. It is a problem that will have ever more serious implications in the future if it is not straightened out.

Space Ex used the data it had to locate an issue. The struts they were using to mount these tanks had basic flaws in some of them that could have caused this accident and they absolutely needed to fix this issue before continuing to launch vehicles. Unfortunately, there is no proof that this issue is what caused the loss of the Falcon 9 during flight. It makes a plausible story and it was definitely an issue that needed to be rectified. However, neither any of the investigators on Space Ex's team that did not work for Space Ex nor the independent NASA investigation team were convinced that this was THE cause of the loss of the Falcon 9 on June 29, 2015.