A Different Type of Union

I grew up in a lower middle class household. Hard work was something to be admired as well as a requirement for getting anywhere in my dad's house. My dad believed that you could be anything or do anything you wanted to do as long as you were willing to work hard enough at it. All of the lessons I learned on basketball courts, baseball diamonds, and football fields could be condensed into that one belief. Practice makes perfect. Hard work and teamwork beats talent and natural ability in the long run every time. Every defeat was met with the same advice; "I guess you'll just have to work harder."

He also put this into practice with chores and work around the house. I worked on cars with him, my brother and I cut yards all over the neighborhood. We painted our house, we put shingles on the roof, and when we moved to the farm we added on to the house, built sheds, chicken houses, fences, hog farrowing houses, and worked on every piece of equipment on the place whenever it broke down. I learned early that hard work is one of the differences between success and failure in most every undertaking. It might not be the only difference, but it was often the only difference we could control. 

I have had a job to go to every day since I was fifteen years old. Before that I had paper routes, firewood to cut and sell, and hay to haul for other local farmers. Hard work was something to be admired in others and required in my brother and I. By the time I had found my way into working as an electrician's helper I had already worked many different jobs. I strongly believed in the work ethic that says you give someone an honest day's work for an honest day's wage. I enjoyed my work and I was good at it. 

I worked my way up from a green helper to a journeyman and then lead man electrician on commercial and industrial job sites in the span of 6 years. Most of the contractors I worked for were willing to let someone take on all the responsibility they were willing to shoulder; as long as they followed through and did good work in a timely manner. I learned by doing for the most part. I asked a lot of questions and I watched every skilled trade on the jobsite. I wanted to learn how everything worked, how all the different trades managed to work together to go from a patch of dirt to a functioning building. 

One thing I learned early on was a disdain for unions. In a strange sort of way unions were responsible for me learning a trade but that's a story for another day. In North Alabama, there were a few union jobsites but they were rather rare. Most of them were TVA plants or federal sites on military bases. There were very few commercial or industrial sites outside of the government that were union in this area; there still aren't. 

Most of the contractors I worked for flat out refused to hire anyone who had ever been in a union. They believed they learned to be lazy, that they worked to quotas that were inefficient and that they were so specialized as to be useless on smaller sites. There is some truth to all of these things. Later in my career, after I went to work for NASA, I came to realize this is still the case for a lot of union contractors but again.... that's a story for another day. 

As a non-union contractor, our wages were significantly lower than union pay scales. The notable exception to this rule was when the non-union contractors managed to win contracts on government facilities or won bids on projects that had Federal funds involved through HUD. On these jobs, we made scale money; which was considerably more than what the going rate for electrical workers outside of these jobs. I was making 11-13 dollars an hour running small commercial and industrial jobs. Scale jobs at the time were paying 17 dollars an hour for journeyman electricians. In other words, the top man on a non-scale job made 4-5 dollars an hour less than every journeyman on a scale job made. 

No non-union contractors paid benefits in those days. No insurance, no vacation, no paid holidays, it was just work until the job ended and hope they managed to win a new bid by that time. In truth, they usually tried to keep their best people so when jobs slowed down the first to go was always the hired journeyman without enough initiative to actually run crews or plan work. I didn't think much about any of this in those days; it was just the way things were. I was proud of my skills and proud to be one of the people that contractors depended on to get the job done. I went quite a few years without ever being laid off. It was a point of pride for me; to be valuable enough that contractors didn't want to lay me off. 

I was gung ho and a hard driver. I finished jobs on time and I did things right, everything had to be square and true and I took a lot of pride in those jobs. I still do today. I can point out buildings all over this area some 30 years later that I worked on. Eventually, I began to notice things that changed my perspective. I saw two people I worked with get permanently disabled because we were in a big enough hurry that we did unsafe things. One of them fell 25 feet onto a concrete floor and ruined both his legs because someone didn't take time to put all the safety pins in a scaffold and it collapsed. One of them cut the main artery in his wrist when a wrench slipped inside a 3000 amp main we were reconnecting on an emergency basis. We had worked 34 hours straight when that occurred. I had to drag him out of the inside of the main before he passed out as we simply coudn't have gotten him out without cutting the panel apart if he had passed out before I could get him out. He never did manual work again. 

As work picked up in the mid-eighties the contractors in this area began to organize. I suppose they were tired of cut-throat bidding each other out of work and stealing each other's men when work was flush. It suddenly became impossible to go from one contractor to another. If you were laid off, another contractor would hire you but they quit competing for hiring each other's help. Wages dropped for all but the very top lead men. I saw this happening but it wasn't affecting me directly as I was never out of work. I had built a good reputation and if I could walk off of one jobsite and onto the next with no problem if I wanted to. I did it several times over people trying to force me to take shortcuts on quality or code violations. I also had a good reputation with all the local inspectors. They trusted my work and knew I didn't do things that were against code so they didn't try to pick my jobs apart. I wasn't going to let a contractor trying to cut corners ruin my reputation and left a couple of jobs accordingly. 

I noticed all of this but didn't really pay too much attention until I was hired to work on a scale job and the contractor was falsifying records for the HUD inspectors. According to his records, I was the only journeyman electrician on that job. This was plainly not true but I figured whatever deal he had worked with everyone else about their wages was between them. Everything was fine until the contractor asked me to sign the progress report that listed all the journeyman electricians as laborers. I refused and told him that the HUD people wouldn’t buy it anyway. He insisted he had already worked it out with the HUD inspector but I just couldn’t bring myself to sign something I knew to be false. The whole scheme soon blew up and the contractor wound up paying everyone on the job as a journeyman along with back pay for all work performed so far. I managed to finish the job but it was not a pleasant work environment and I couldn’t wait to move on to a new contractor.

Soon after that job I was told that I needed to get an ABC certification as a journeyman electrician to continue working. ABC stood for Associated Building Contractors. I soon realized why wages were falling as it was in reality a union for people who owned electrical contracting companies in the area. Under the guise of improving the trade, the owners had organized a union just for owners and they were steadily knocking down wages by doing so. The workers who had helped private contractors keep unions off of local job sites suddenly found themselves victimized by a different type of union; a union for owners.

The ABC association soon was all over the state. As more owners joined, they lobbied Congress to lower federal scale wages for construction electricians. When the next wage report came out federal scale wages were down considerably. When I started doing electrical work it was around 17 dollars an hour. Two years after the ABC association appeared on the scene it was down to 10.80 an hour. The only people that were asked about prevailing wages were the owners of the companies so it isn’t too hard to understand how a union for owners can control that information.

I soon found the only contractor left in town who wasn’t a member of the ABC association. Shortly after that I decided to find a job with some health insurance and benefits. This meant a job in maintenance instead of construction but I was skilled enough to make that jump rather easily.

It was almost ten more years after the appearance of the ABC association before scale wages came back up to 17 dollars an hour. By then, federal scale wages were lower than what contractors on the street were having to pay to get people to work for them.

Now…. in this area, you will be hard pressed to find a construction electrician who isn’t an illegal immigrant; at least you won’t find many of them. For that matter you won’t find any skilled trades people in construction besides the top lead men who are not illegal immigrants. The ABC association long ago disappeared. There’s really no need of a union for owners to hold wages down when you can simply hire illegal immigrants for 15 dollars an hour. People who are afraid of being deported also aren’t very likely to ask for a raise or any semblance of benefits either.

I think about all of this when I read an article claiming that young people need to learn a trade; that demand for skilled trades people is climbing. Perhaps if Trump actually does stop illegal immigration this might be true but it isn’t at the moment. Of course, if it does; we can always crank the ABC association or something like it back up.

I recently saw a local post on Facebook suggesting that Trump’s election would introduce the rest of the country to Alabama Economics. Good luck with that……

The Truth

Truth- that which is in accordance with fact or reality

In the house I was raised in truth was unequivocal. It was not something that there was room to argue about, there was no wiggle room around truth. You either told the truth to my father or paid the consequences. It was one of his hard and fast rules that formed the foundation or our relationship with him. With truth came trust and acceptance in his company, anything else was simply unacceptable. I don't remember my first exposure to this rule of law but I do remember that it was universally applied. If we got in trouble for something he would ask us what we had done. Woe be unto whomever strayed from the truth when this question was asked. I learned very early that his wrath for doing the wrong thing was inconsequential compared with his wrath for not telling the truth about it later. Even though his wrath could be severe, it wasn't so much his wrath that I was afraid of as it was the loss of his respect. Respect was hard earned from my dad but it was also more valuable in his estimation than anything else he gave us and we all realized this at a young age and valued it accordingly.

I have carried that respect for truth throughout my life. I don't mind a good story now and then, and good stories are often flavored with what we want to remember just as much as they are with what actually happened but when it comes to human interaction and the serious business of work, truth is absolutely necessary if we are to achieve anything in this world. We simply can't learn lessons about anything unless we learn to respect the truth first. Everyone makes mistakes and I believe we actually learn more from our mistakes than our successes. However, it becomes impossible to learn anything from our mistakes if we refuse to acknowledge them. We are simply doomed to repeat them endlessly at that point, never advancing beyond them because we don't understand they are mistakes.

The current presidential election has presented us with two candidates who have no respect at all for the truth. Trump is a serial liar of enormous magnitude. He regularly denies saying or doing things that he has been publicly recorded saying and doing. I have known people like Trump before, habitual liars are not a new invention of this political campaign, they have been with us for a long time. Trump seems to be the most obvious of this particular characteristic that we have had run for president but he is not the first liar we have had run for this or any other office. Politicians who say one thing and then do something else are quite common. The artful use of language to give the implication of one intent while doing something completely opposed to this intention seems to be part of holding public office for a lot of people.

Trump takes his dishonesty to levels that we haven't seen in someone running for president. The airwaves are literally full of overlays of him first saying something and then blatantly denying having ever said it. He has a childish tendency to simply deny his own statements endlessly even when confronted with the proof of him saying it. It is quite remarkable to see actually. I am reminded of a conversation I had with one of my sister-in-laws concerning her child's refusal to admit he had done wrong. I told her he must be forced to admit the truth before he would ever become capable of being trustworthy. Evidently, Trump has never been forced to be truthful which leads to the fact that he is bascially unworthy of anyone's trust as the multitude of lawsuits he has been involved in are open testament to.

Unfortunately, Hillary Clinton also seems to have little or no respect for the truth. Her career is full of half truths, tortured attempts at equivocation and twisted logic to justify actions that may have been honest mistakes at the time but now are seen as political baggage and disavowed accordingly. She seems incapable of simply admitting that she made a mistake and moving on. Every change of position, every twisting of statements by others to point them towards meanings that they did not have all add up to someone who has no basic respect for the truth. The truth is that opinions evolve, people change and political positions evolve as well. Unfortunately, very few politicians have the temerity to point this out when their positions evolve.

The Clinton's have a very long history in US politics. It would be impossible for them to hold the same positions they did in 1980 and still be electable in 2016 because the US voting population has evolved dramatically since that time. I would have more respect for Hillary Clinton if she were to point this out and admit that neither she nor her husband are infallible. Unfortunately, as we saw in the Lewinsky case, Bill Clinton has no real respect for the truth either.

I suspect we all know habitual liars. It is a fact of life that we will be confronted with them at work, in our personal lives, and at various social functions. I try to avoid them as much as possible. I simply have no respect or use for them. No one is perfect and we all suffer from various personal weaknesses and flaws but someone who has no respect for the truth has an extremely low value as a human in my estimation for the simple reason that they are permanently frozen in a state of perpetual childish tendencies that they have no hope of advancing past because they have no understanding that they should. Habitual liars never admit or their own mistakes, therefore they never understand them.

It disheartens me to think that Hillary Clinton and Donald Trump are the only two real choices we have running for the highest office in the land this year. One of them is provably a habitual liar of monumental proportions. The other has shown a very haughty and public disdain for the truth on numerous occasions. Quite honestly, neither of them would be a guest I would choose to entertain in my home, much less someone I would choose to run the most powerful nation on earth.

Aging not so Gracefully

I will be 57 very soon. I find that I am not aging as gracefully as I might like. I don't like the fact that I have trouble recalling people's names, specific nomenclature for tools and techniques, and generally a lot of things that used to be always easily pulled out of my memory banks that now get exceedingly slow. This worried me enough that I read several books on how aging affects the brain. I felt better after discovering that the brain compensates in other ways for this lack of speed, but it still bothers me to have a conversation constantly interrupted with pauses and a lot of "I can't think of the name right now" comments.

I have also discovered that my writing often takes on unintended comical aspects. I sometimes simply leave letters off of words as if my brain can't be bothered to complete the whole word. I regularly substitute on and in within sentences as well as this and these although these comes out "thes". My proofreading has always been terrible as I tend to read what I meant to write instead of what I actually recorded. This makes my new tendencies even more frustrating.

I have a routine I go through when I leave work to check that I have my badge to get back into work with and my cell phone in my pocket. I have twice gotten out of the car and started back into the shop to retrieve my missing cell phone WHILE I was talking to someone on the "missing" cell phone. I have even more elaborate routines while leaving home for work that involve checking and rechecking whether the stove is off or the water faucet I use to fill the dog's bowl is off. You don't even want to know what I go through when leaving on a trip. None of this used to happen when I was younger.

Quite a few years ago I figured out that I had to work out regularly to maintain any semblance of physical strength so I do that fairly routinely. Still, rigorous exercise produces levels of pain in the mornings that sometimes makes me think I should just retire permanently to the couch. Back pain, shoulder pain, ankle pain, and pain in my hands from a lifetime of manual labor and a youth of physical confrontation are constant companions. I don't enjoy these companions but I am never quite free from them either. Concentration on the task at hand and a certain sense of creative enjoyment dulls them considerably during activity, but they soon come back with a vengeance when I sit down for a while.

Some time ago, I worked with a somewhat crotchety retired guy who had come back to double dip on the system. He came in one Monday morning limping considerably. I asked him what was wrong and he whirled around pointing his finger at me accusingly, "Getting old is not for p%^^#s. You'll find that out one day." Wherever you are today Chester..... I couldn't agree more. Still.... it beats the only other alternative we have.

A Bridge Too Far....

A bridge too far is simply an act of overreach; a step too far in advance of capability. In psychological terms it is a symptom of hubris or an over inflated sense of achievement that leads to disaster. At this point at least, it appears that Elon Musk and Space Ex are victims of their own bridge too far. Success in returning rockets and relanding them have been fantastic and extremely impressive but they seem to have led to an attitude the skips over the careful understanding of process in search of further achievement.

I will once again point out that all of the data is not in yet on what happened to cause the Falcon 9 to explode on the pad at Cape Canaveral during loading. While it appears that a COPV in the second stage LOX tank exploded and caused the destruction of the rocket itself; the investigation is ongoing.

In my last post, I went into some detail about the fragile nature of COPV's as far as mechanical external damage is concerned. I can't adequately express how important to the health of a COPV that basic handing integrity happens to be. While COPV's can and have been designed for massive overpressurization limits because of the lack of weight associated with more composite wraps, they are also extremely vulnerable to external damage that might immediately seem inconsequential, but can quickly lead to the failure of the COPV.

I have also pointed out that Space Ex didn't seem adequately concerned with handling of COPV's, both with the mounting methods they use to put them inside the LOX tanks and by the lack of understanding of some of their personnel that they are not meant to be stood on. For whatever reason neither the person standing on the tanks, nor the person distributing pictures documenting this act seemed aware that this might be a problem which of course leads to the question of how many other people in the handling chain for these tanks at Space Ex were similarly unaware.

While this is troubling, it is hardly proof that it caused either or both of the recent Falcon 9 explosions. As I pointed out in my last post, the sequence of events if that is what happened does seem to match the available data. There are a lot of other possibilities however. I will try to briefly explain some other things that could have quite possibly caused an inadvertant rupture of one of these same COPV's.

Let me go over another possible scenario that could have led to the failure of one of these COPV's well below its rated pressure. There are a couple of physics theories that have to be explained in order to understand the following sequence of events. The first one is something called the Joule-Thomsen effect. What Joule-Thomsen says is that expanding a compressed gas across an orifice predictably changes its temperature. This is largely the basic theory that led to the discovery of modern day air conditioning. Most gases at or near ambient temperature will tend to cool as they expand across an orifice. Therefore, taking a compressed gas such as Freon and expanding it across an orifice or valve will chill that gas quite dramatically. This cooling effect is exactly how a modern day air conditioner works. Compressed freon is expanded across an orifice into a cold tubing bundle. If air is forced across this bundle by means of a fan, the air itself is chilled accordingly. Air conditioners are a closed loop system. This means that the same Freon gas is later collected and forced through a compressor which is used to pressurize it again so that it can repeat this basic process again.

There is a rather odd anomaly surrounding three gases when it comes to the Joule-Thomsen effect. Hydrogen, Helium, and Neon have what is called a negative Joule-Thomsen effect under certain conditions. In other words, they can produce heat when expanded across an orifice, dependent on starting temperature of the gas and pressure. In the case of Helium space vehicle pressurization systems tend to cross over and through this barrier as a normal event. In other words, part of the time that you are transferring helium through a valve or orifice it will get colder and then it will go through a stage where it produces heat and then back to colder. It is predictable, if not easily calculated but it is something that should always be taken into effect when pressure rating helium and hydrogen systems. Higher heat inevitably leads to lower pressure ratings for containment vessels, valves, and pipelines in such systems.

The other side of this same process then has an effect. It is called adiabatic heating. Adiabatic heating is simply heat of work that is added to the Freon as it is stepped up to a higher pressure again by the compressor. This heat is usually dissipated in large coils that are exposed to circulating air which tends to cool the gas by the time it gets back to the orifice to be expanded again.

Another example of Adiabatic heating is a diesel engine. Diesel engines don't require spark plugs to light the fuel in each cylinder because they relay on the adiabatic heating process that occurs when the inlet valve is closed and the piston is driven at a high rate of speed inside the cylinder to compress the air/diesel mixture of gas in the cylinder. The adiabatic heating combined with the piston still being hot from the last stroke is high enough to light this mixture causing an explosion which forces the piston back down. This in turn causes the main crank to turn and push another piston up with the same result.

Adiabatic heating can be both large and fast, dependent on the gas being compressed and its volume. It is therefore also an ignition mechanism in fluid systems that has to be carefully considered when transferring gases at high pressures and volumes. Extreme heat weakens pressure ratings because the metals themselves lose strength and become more ductile when heated. This difference in tensile strength can be very large, depending on material, rate of heating, and purity or metallurgical control of manufacturing process.

As I discussed earlier one of the relative oddities of COPV's is that they will actually contain higher pressures at cryogen temperatures. This is because the extreme cold at tends to shrink the composite fibers more than the internal aluminum tank. This same quality reverses as a COPV is heated. The aluminum tank tends to expand to stress the carbon fibers more, putting them in a preload condition that limits the expansion they can take as the inner tank starts to try to expand with increases in internal pressure. If a COPV is slowly and uniformly heated this effect is lessened. If it is heated in a non-uniform matter it increases.

When we have used COPV's in testing Shuttle systems we routinely monitored internal temperatures in them to guard against this effect. Very minor increases in temperature inside the tank while the outside carbon fibers remain at the same temperature were deemed problematic enough that we had cutoffs that stopped the pressurization process automatically if we gained over 120 degrees F on the inside of the COPV. I know from experience that this was easily done with a very small volume pnuematic system so that we would take a very long time to charge the COPV's to operating temperature to do our tests.

These numbers were based upon a basic 50 degree F differential between the inner part of the COPV and the outer carbon fibers since we were operating at a roughly 70 degree ambient environment. It is routine practice to monitor internal temperatures on COPV's as they are pressurized so that you don't effectively make them able to withstand less pressure. From what I have been able to gather, Space Ex also monitors inlet pressure while filling the COPV's on Falcon 9. What I don't know is how they are monitoring this temperature.

Let me explain why this is also important. In the first place, the most common temperature measurements are made by utilizing a thermocouple which is simply a connection of two dissimilar metals. These dissimilar metals produce a voltage at their junction that changes with temperature. By monitoring these changes it is possible to accurately read this voltage and infer a temperature.

There are other instruments that are used to make such measurements but this is the most common and least expensive as well as being the fastest at monitoring temperature changes so I am somewhat giving Space Ex the benefit of the doubt in assuming they have an accurate measure of internal temperature on the COPV's as they fill them. When I say fast, I mean the amount of time it takes to actually reflect a change in temperature. In the data aquisition world temperatures are amongst the slowest measurements for the simple reason that even the fastest instruments for temperature measurement are rather slow. A step response of 200 milliseconds on the fastest thermocouple would be very good. In other words, dependent on instrument and data rate it is very hard to see rapid temperature changes.

Since Space Ex is filling several COPV's at the same time I am sure they are utilizing a temperature sensor on a tube instead of individual temperature sensors inside each COPV. This matters because the relatively large volume of a COPV inside of a tank filled with LOX (-297 Degrees F) will necessarily read a much lower temperature as the small area inside the tube will be more readily chilled by the LOX than the volume inside the COPV. In essence the temperature monitoring instrument they are using on the COPV's as they fill them is relatively slow to react to changes and physically masked by the physics of its mechanical location.

Let's also keep in mind that the physical environment of the COPV's is greatly exacerbating the relative weakening of its ability to hold high pressure gas. The COPV is seeing at least -297 Degrees F on its carbon fiber exterior at the same time that helium being heated by adiabatic heating and negative Joule-Thomsen effect is being introduced into its interior. We are probably talking a 400 degree F temperature differential that it is routinely seeing on Space Ex's rapid fill propellant transfer which will necessarily vastly lower its pressure rating. Depending on how high this differential actually is, it could rather easily exceed the safety margin of the COPV's pressure rating.

This is the part where we begin to get into the "Bridge too Far" analogy. Space Ex and Elon Musk have their eye on much bigger things than the resupply of Space Station that Falcon 9 is specifically tasked with performing. Musk has made no secret that his eventual goal is the colonization of Mars and they are as I write this developing heavier launch bigger vehicles to reach this goal. It is this ultimate goal that is causing them to push the envelope so hard on Falcon 9. There has been a lot of publicity about reusable rockets and a very public "competition" between Space Ex and Blue Horizon to return rockets and land them successfully in order to keep costs low on launching payloads.

This requires extra fuel and oxidizers to be loaded onto Falcon 9 in order to have the requisite power to control its landing process manuevers. To their great credit both Space Ex and Blue Horizon have managed to be very successful in recovering their rockets with such manuevers. Since this is not what Falcon 9 was originally designed or contracted to do it is has been a little problematic for Space Ex to find ways to get the extra fuel and oxidizer on board to carry out these manuevers and still deliver its payload to Space Station.

One of the ways they have managed to do this is to steadily increase the supercooled effect of the LOX they use on Falcon 9. Supercooling of LOX is not a new idea either, as NASA has used this idea to increase turbopump efficiency for quite a few years. What it involves is decreasing the temperature of LOX away from its boiling point and towards it freezing point. In other words, the colder they can get their LOX before launch, the more energy it will contain. This effectively gives them the margin they need to carry out the extra manuevers necessary to safely land the rockets on their return descent to Earth.

One of the main factors for having super cooled LOX is to load it quickly and to make sure it does not sit in the LOX tank for long before launch. The longer it sits in the tank the more it tends to stratify which leads to less efficient burns and more LOX expended on ascent. Space Ex has been constantly speeding up its rapid load process for quite some time now. This means that massive amounts of LOX and RP1 are transferred to the vehicle just 30 minutes before launch. It also means that Space Ex must launch immediately after loading so that all other checkouts and final preps must occur after loading propellants.

The latest load of propellants as a dry run test was the most rapid load they have yet attempted. This effectively means that the LOX in the tank was colder than it has ever been on a propellant load. It also means that the Helium fill which must occur after LOX is loaded to take advantage of the increased density that putting the COPV's in the LOX tank in the first place was necessitated by. In their every increasing haste to cut the time that LOX sits in the vehicle I am sure they are also accelerating the rate at which they fill their COPV's which also increases the heat load by adiabatic heating and negative Joule-Thomsen effect.

In other words, for reasons completely outside of their actual manifest task, Space Ex has neccessarily vastly increased the temperature differential their COPV's see on propellant load. Is this relationship the key to the latest loss of a Falcon 9 on the pad at Cape Canaveral? I guess time will tell if this is actually a bridge too far or just another bump in the road for Musks' frenzied path to Mars.

The Fickle Nature of COPVs

As I have already discussed, COPV's are a vast improvement in weight vs. functionality in that quite high pressure COPV's can be manufactured that weigh a small percentage of the heavy wall tank needed to safely contain high pressure gases. When you start adding in extreme temperatures such as LOX (-297 Degrees F) then the metal tanks must also be of high purity stainless steel as well, which gets even more expensive and complicated to design and build.

One of the first groups to work with designing and building COPV's worked at Marshall Space Flight Center in Huntsville. One designer in particular spent much of his career working on developing processes to accurately design and build COPV's rated for very high pressures at very low temperatures. While COPV's have been around for a while the idea of using them inside of cryogen storage tanks was an undertaking that required a lot of research and development. What kind of fibers to use, what kind of helical winding pattern to overlay the tank in layers, what kind of glue to use and how to cure it; all of these factors and many more were relatively unknown in the beginning simply because no one had experience in building these types of tanks.

As luck would have it this designer happened to get some research and development money just about the time that our small testing lab was getting started. When he approached our group about doing failure testing on his COPV's we were able to give him some pretty low bids on doing this testing so it was a marriage of convenience for us both. He needed an inexpensive way to do failure testing on his designs and we needed funds enough to take on the relatively small amount of work this would entail at a low rate.

The first tests we did on these bottles involved hydro testing them to failure. The process involved putting them in a hydro chamber which we would then fill with water to help contain the blast wave. Then, we would fill the test bottle with water and begin pumping it up to its failure pressure. Hydro testing is much preferred to pneumo testing because water is not compressible. Therefore, when the bottle ruptures the pressure is quickly abated with a minimal shock wave as the water escapes the ruptured bottle. When you do pnuemo testing you pressurize the bottle with a gas which is very compressible. As the gas pressure increases until the tank ruptures there is quite a violent release of the stored energy in the gas as it has to completely expand back to its original state.

It is the difference between filling a balloon with air and puncturing it with a pin and filling the same balloon with water and puncturing it with a pin. The balloon filled with air will violently fail as the compressed air escapes the tiny hole and rapidly expands. The balloon filled with water will leak slightly and slowly relieve the rest of the pressure through the hole the pin made. There is no stored energy in the water because it is not compressed.

We have a heavy duty hydro chamber with blast proof glass on top so that we could film the test as well. We used a digital data recording system to record the water pressure at a high rate of speed so that we could see the exact pressure that the bottle would rupture or fail at. Unlike the water balloon analogy, we were taking these bottles to 3000-5000 PSIG before they would rupture so it was a little more violent than pricking a water balloon with a pin but still much less violent than doing the same failure with an expanding gas.

The point of the testing was to prove that the COPV designer's processes were controlled well enough so that a series of bottles manufactured with the same process would withstand the same pressure before failing. It was the first step in figuring out safety factors for the COPV's that he was building. The first tests were immediate successes. Not only were the failures all above predicted pressure, but they were very consistent as well. We tested some 20 bottles to failure at 3500 PSIG and they all failed within 100 PSIG of each other which was actually much better consistency than anyone was predicting.

What wasn't really predicted was the way in which the COPV's failed. Since we were trying to get exact data we were pumping the bottles up fairly slowly. The first bottle we ruptured exhibited some strange behavior we were not expecting. As we approached the predicted failure rate and slowed our pumping rate even more we heard a muffled popping noise followed by an immediate drop in internal pressure in the bottle. We were puzzled by this to say the least. The pressure stabilized but it had dropped some 75 PSIG immediately when we heard the popping noise. As we began pumping again we would see the pressure rise again but it would immediately fall back whenever the pumping piston retracted. Having done a lot of hydro testing over the years we suspected we were seeing some sort of tiny leak to cause the pressure to drop. Since water is not compressible even a small drop of water is leaking is enough to cause a significant and measurable drop in pressure. What didn't make sense is why the pressure stabilized again after it dropped.

We surmised that we may have found the pressure at which a thread or fitting was leaking but since the tank was under water there was no way to locate where it might be leaking. With the designers permission was decided to go ahead and see if we could overcome this leak with enough pumping pressure to cause the tank to fail. It took quite a few strokes from our pumping system and we had several more muffled pops followed by further drops in pressure but we eventually did rupture the bottle pretty violently. Once the carbon fibers gave way and the tank ruptured it looked very much like an exploded bundle of carbon wires. The aluminum tank underneath ripped violently open and frayed carbon resembling an angry porcupine angled sharply out from the breach in the tank.

It made interesting viewing on video and the data system captured the exact peak of pressure that caused the tank to rupture. As we looked at the pressure data more carefully later we could see the rises in pressure followed by the gradual drops after the popping noises started. We soon realized that the popping noises we were hearing were the individual carbon fiber strands popping in the bottom layers of the wrap. Each time a strand broke the aluminum tank swelled a little more as it was freed from the captured restriction of the composite overwrap material. We were effectively blowing the aluminum tank up like a balloon as the carbon fibers failed, adding space for more non-compressible water with each breakage. Eventually, enough carbon fibers failed so that they could no longer contain the swelling aluminum tank and the whole tank violently ruptured.

Later on when we filmed with high speed video we could actually see the tank lurch with each pop of a carbon fiber strand but by then we were well familiar with the failure mechanism of the tanks. The underlying carbon fibers break first and since there are so many layers of fibers it is quite impossible to see any change in the tank but the pressure trace sees the extra volume afforded by the resultant expansion in the form of decreased pressure.

After we completed a series of tests of water testing the designer suggested he needed to know how cryogen temperatures would affect the strength of the tanks. In other words, since we knew he could consistently predict their failure in water could he also consistenly produce the same results at cryogen temperatures? The end result of such design and testing would be to have a COPV that could be imbedded in a LOX or Liquid Hydrogen tank. The weight saving would be huge and the expense to produce such COPV's would be much less than a similar metal tank.

Rupturing a COPV at 3500 PSIG in Liquid Nitrogen (-320 degrees F) turned out to be a little more problematic than anyone planned. The first need was to get the tank chilled to LN2 temperature which took quite a bit of LN2 and was a slow process involving creating a tube that the COPV would fit inside. The tube would then be slowly filled with LN2 to chill the outer part of the tank to temperature. After this was accomplished we would fill the inner part of the tank with LN2, being careful to remove all compressible gas at the same time. This was accomplished by a high point bleed that we opened until we got LN2 out as we filled the tank from the bottom.

It is important to remove all compressible gas to minimize the stored energy that will be released when the bottle ruptures. While LN2 is like water not compressible we knew it would go through a rapid phase change once the tank ruptured. At 68 degrees F LN2 expands 694-1 as it changes for a liquid to a gas. In other words one gallon of LN2 instantly increases to the volume of 694 gallons when this phase change occurs. This phase change is almost instantaneous so we knew that when the tank ruptured we could see a very violent and quick phase change shock wave.

To minimize the already considerable explosive power we were going to produce we were very careful to keep the LN2 in the bottle in a liquid state. We knew that any LN2 that flashed to gas would then be compressible, thereby increasing the explosive power we were going to release when the tank ruptured. We performed this test in the abandoned back area of the test area at Marshall. We utilized some very large steel I beams to build a barrier around the test setup knowing we could direct the shock wave upward in this manner. We also ran all our control and instrumentation wires into a blast bunker on the bottom of one of the test stands so that we would be removed from the vicinity when the rupture occurred.

We put temperature sensors on the pump feed line into the COPV and planned to keep our pumping speed low enough so that the natural heat of compression of a pumping piston would not flash the LN2 to a gas as we pushed it at increased pressures into the COPV. We set up video to capture the explosion itself but the main data we were after was the pressure at which the COPV would rupture at LN2 temperature. The designer suggested it might actually hold more pressure at cryogen temperature as the carbon wrap fibers themselves would tend to shrink and more tightly hold the inner aluminum tank in compression.

As soon as everything was set and we had cleared the surrounding area of all personnel we began our process. It took quite a while to slowly chill the COPV so that we could cover it with LN2. The real problem came after we filled the COPV with LN2 and began slowly pumping the pressure up with a cryogen pumping cart. The heat of compression would quickly overcome the boiling point of LN2 and we would begin to flash to gas on the inlet line of the COPV. Our test design review board had set a hard temperature number barrier on this line that we could not go above as it would increase the explosive power of the tank failure considerably.

After several hours of pumping we were nowhere near the pressure we thought it would take to fail the tank because we were having to stop so frequently to allow things to chill back to liquid temperature. Unfortunately, every time we stopped the return to liquid temperature would also decrease the pressure in the COPV as the density dropped. It was a losing battle and we soon knew we couldn't gain enough pressure to fail the tank.

After more study we decided to better insulate the fill lines and move the cryogen pump much closer to the test article. We moved the massive steel I beams with a crane to get everything closer and set up for another run at failing the COPV. We improved the process such that we could get closer to the pressure we were looking to fail the tank but still eventually hit a point where we could no longer gain pressure and keep everything at liquid temperatures. The next step would have been to include vacuum jacketed lines and a lot of expense that no one had funds for so after a quick phone session with the COPV designer and our test design review board everyone concluded we would let the temperature creep up as much as needed to achieve rupture pressure. The designer needed data for a conference he had coming up and since we had cleared the test area of personnel we simply bought the risk of destroying some of our test equipment when the tank ruptured.

We knew both the liquid tube the COPV was chilling in and all of our safety barriers would force the blast wave upward when the tank ruptured so we were fairly certain that we wouldn't do a lot of damage to anything besides the tube and some of the attached tubing and instrumentation lines. Once we got underway again we got back to pressure fairly quickly and then began speeding up the pumping process as we watched the temperature and the pressure in the COPV climb. We knew we were creating a compressible gas bubble in the COPV to add to the phase change explosion that was coming but everyone had agreed it was an unavoidable risk if we were to meet schedule and budget.

The tank, true to the designer's suggestion, actually ruptured at some 400 PSIG higher than the same design had failed at in water. We got a beautiful pressure trace showing the same popping and swelling scenario we had seen in water. We also got exactly two frames of video on our normal speed video showing a veritable rocket rising on a plume of cold gas out of the liquid soak tube. It took a little while to find the remains of the COPV and some of our tubing still attached to it. It was some 200 yards away in a swampy area next to the barrier fence seperating the test area from a wildlife refuge.

It was quite an impressive audible blast. The same size and design bottles that we had been more or less harmlessly popping underwater had produced a titanic blast with the phase change and added compressible gas that filled the bottom 1/4 of the tank when it ruptured. We don't really know how high it went as it went out of camera view in two frames.

We later did similar destructive testing at Liquid Helium temperatures to simulate the pressure rating of a COPV in Liquid Hydrogen. The design and control process that the designer used in making these tanks was very good as all failures were both predictable and consistent across many samples. This designer later left NASA and branched out to form his own company that now produces and sells these COPV's to space flight companies. The only company that I know of that utilizes these COPV's in cryogen tanks on vehicles is Space Ex.

Going back to the data that seemed counterintuitive on the Falcon 9 that exploded mid-flight; the confusing thing for Space Ex and many others that looked at this data was that the Helium pressure in these COPV's dropped at the same time that accelerometers picked up a popping "sound" in the area of the COPVs. It then immediately returned back to "normal" range right before the LOX tank overpressurized and exploded. As I have detailed above this is exactly the same sequence we saw when we were testing these tanks to failure in our lab. The popping noise was the inner carbon fiber strands breaking. The drop in pressure was the aluminum inner tank expanding like a balloon.

On Space Ex's Falcon 9 there are several COPV's tied together with a common tubing manifold. If one tank were to start to swell as the composite fibers break the pressure would quickly equalize between it and the other tanks creating a return to "normal" pressure reading on the whole system. However, the increasing pressure would quickly break more composite fibers until the damaged tank failed explosively. The severity of the resulting expansion would be maximized by the fact that the LOX tank itself was very full at this stage as the second stage was not in operation. This would mean there was a very small ullage or gas bubble for compression in the tank leaving only incompressible LOX which would quickly rupture the LOX tank.

Both Space Ex and the supplier of the COPV's will quickly explain that the COPV's have been thoroughly tested to much higher pressures than those they saw on flight. I will also attest to the fact that the designer of these COPV's has extensive data showing how good his design and manufacturing control processes on these bottles are. I have no doubt that these bottles will not fail under design pressure when handled properly.

Going back to my original discussion of COPV's I also explained that COPV's suffer from a couple of weaknesses as flight pressure tanks. The first is cycle limits, although this is probably not a problem in this usage. The second is the fragile nature of the exposed carbon fiber shells themselves. Each small carbon fiber is glued and interlocked in an intricate helical pattern to wrap and contain the thin aluminum tank underneath. Since the weight of the carbon fibers is minimal it is both cost effective and weight efficient to build in large overpressure ratings for COPV's. A tank that much contain 5500 PSIG can easily be manufactured to safely contain much higher pressures before failing. I have no doubt that the tanks flying on Falcon 9 are indeed rated and tested to much higher pressures at LOX temperatures than those they actually see in flight.

Musk has been especially adamant that the Carbon shells will not fail at pressure. No doubt he has seen a lot of data from the manufacturer proving this to be true. However, the fragile nature of the carbon fibers themselves require very careful handling from manufacture to test to installation to assure that they are not damaged before usage. Unfortunately, there is evidence that this has not been the case in the usage of these tanks.

Previous to flight Space Ex is in the habit of taking photographic evidence of all phases of their Falcon 9 assembly process. These pictures are known as "closeout" photos and stand as visual proof that all nuts are lockwired and all tiedown and cabling is carefully attached in the vehicle. I have not personally seen these pictures but some of the review teams I work with have. In at least one of these pictures the photographer himself is standing on the support struts that the COPV's are mounted to. No flight hardware is rated to be used as a standing platform. It is an egregiously bad practice for anyone to do so under any circumstance; either during fabrication and assembly or at any other time. The fact that someone is indeed seen standing on something as delicate as the COPVs which are inherently prone to serious and debilitating damage from relatively minor exterior mechanical force is inexcusable. The fact that people within Space Ex have seen and distributed these pictures seems to point to the fact that there is little or no understanding of the fragile nature of the COPVs.

While this is probably a little long winded it does plausibly match the actual data from the Falcon 9 that exploded in mid-flight. Could the same thing have happened to the Falcon 9 on the pad at Kennedy. Preliminary evidence suggests that it was a COPV failure in the exact same location that led to this explosion as well. There are also some necessary precautions that must be taken when initially pressurizing COPVs with gas that are not necessary with metallic tanks. I will go into that in my next post.

For Want of a Nail.....

There is an old parable: For want of a nail the shoe was lost;
For want of a shoe the horse was lost;
For want of a horse the battle was lost;
For the failure of battle the kingdom was lost—All for the want of a horse-shoe nail.

It seems to quite accurately describe the version of events the Space Ex has officially released to explain the loss of its Falcon 9 in mid flight. Space Ex believes that a bolt failed on a strut assembly that holds down the Ghe COPV's inside of the second stage LOX tank. The failure of one of these bolts set off a chain event that within one second resulted in the overpressurization and explosion of the second stage LOX tank and destroyed the Falcon 9.

After Space Ex was able to triangulate high speed accelerometer data and locate the original "sound" that corresponded in time with the loss of Ghe pressure in the COPV's inside the second stage LOX tank they quickly began conducting tensile strength tests on strut assemblies they had in stock. Although some of these strut assemblies failed at around 6000 pounds of force instead of the rated 10,000 pounds of the strut it still didn't explain how one could have failed at 2000 pounds of force which is the calculated load on the struts and maximum G force during ascent.

Eventually, Space Ex was able to find a bolt for one of these assemblies that failed at 2000 pounds of force. Therefore, they concluded that it was probably a bolt that holds the strut assembly to the tank that actually failed. During subsequent press conferences Elon Musk went into a lot of detail explaining how the strut assembly was not manufactured by Space Ex but was instead specified and bought based upon this manufacturers strength specifications. When pressed by reporters to name the manufacturer Musk declined and explained that it would not help the situation. Musk went on to explain that further metallurgical testing on the bolt itself showed improper grain forging which could have also led to a similar failure on the Falcon 9 that exploded. He seemed to have found the smoking gun for the failure sequence that led to the loss of his rocket.

When asked about the materials that the assembly was made of he also declined to go into details but put forward the information that it was a type of steel and was rated for a much higher failure pressure than it would have ever seen on flight. This is very curious, especially when in the same interview he went on to explain that they were considering going to an inconel material which is much harder to come by and astronomically more expensive to buy and manufacture. In the short term he suggested that they didn't actually test the strut assemblies previous to the loss of this vehicle but would certainly do so in the future. He was careful to explain that they had used the manufacturer's strength rating in lieu of actual testing but that at least one improperly forged bolt had been found in their inventory.

A "type of steel" is a red flag for anyone familiar with cryogenic atmospheres. Carbon steel materials lose all of their tensile strength when exposed to cryogenic temperatures. The immediate embrittlement that such temperatures cause in the granular structure of carbon steel is instantaneous and catostrophic. Just as an example, we experienced a catastrophic failure on a high pressure pipeline because of very short term exposure to LN2 (-320 degrees F). We had a heat exchanger controller failure that led to a small amount of LN2 trickling into the 3" 6000 PSIG pipeline previous to starting pumps to pressurize the system. The pipeline soon violently exploded at less than 200 PSIG in a location where it was in mechanical tension.

Even stainless steels with their lower proportion of carbon lose proportional amounts of tensile strength when exposed to cryogenic temperatures which is why careful control of metallurgy, forging process, and purity is required in all such systems. Different grades of stainless (depending largely on the amount of carbon to nickel composite) have different reactions to such temperatures. What has since become public knowledge is that Space Ex was buying off the shelf strut assemblies from a manufacturer that was not aware of the environment they were to be used in. Space Ex took some standard temperature de-rating tables based upon the assumed metallurgy of the struts they were buying and effectively load rated their strut system by analysis. There is nothing wrong with load rating by analysis as long as strict compliance of material pedigrees are observed but that does not appear to be what happened in this case. Since the manufacturer was not aware that these strut assemblies would be exposed to cryogen temperatures they do not track forging processes and metallurgy necessary to do such de-rating by analysis. This also explains why ultra weight conscious rocket ship designers used 10000 psi rated struts in a 2000 psi application.

As I have already mentioned in this thread, Space Ex has long struggled with configuration control of the hardware they are flying. It is a byproduct of being in a tremendous hurry to launch payloads. To be fair Space Ex is driven to this frenzy by the government that is punishing them monetarily for delays on manifests. Add to that the open competition they find themselves in to gain more launch manifests and you begin to see the extreme pressure they are under to launch vehicles. Under such pressure there is little wonder that they took the shortcut of buying off the shelf hardware and downrating it by analysis. It doesn't make it a good practice but it is understandable.

It doesn't make Musk's claim that a bolt "snuck" through the system accurate. It may have, but the real fault was in using assemblies in environments they were not made to be used in without at least explaining to the supplier that they needed to control the metallurgy and purity to make Space Ex's analysis hold up.

Unfortunately for Space Ex there is no proof that any of this is what caused the loss of the Falcon 9 mid-flight. While such a sequence of events seems to fit most of the data that Space Ex has, there are also parts of this story that do not fit the data. This is why neither NASA's independent investigation signed off on this theory, nor did any of the members of Space Ex's investigation that do not directly work for Space Ex. The broken strut assembly scenario is one of several fault tree sequences that could explain what happened. However, there is at least on major piece of data that most definitely is not explainable by this scenario.

The first actual warning that something was amiss on the Falcon launch was a minor drop in pressure on the Ghe pressure of the COPV's mounted in the second stage LOX tank. Musk's first public statements about the investigation brought out the fact that the data seemed counterintuitive. The data, which was taken at a relatively low rate of speed, showed a drop in pressure followed by a return to "normal" pressure. The transducer that monitored this pressure was mounted on a manifold assembly tied to several of the COPV's in the LOX tank. Through the years of doing this type of work I have been exposed to several instances where the data doesn't seem to match any logical sequence of events. It is frustrating to say the least. At some point it is not unusual in such a case to believe that you may have just found THE exception to the universal laws of physics. Reason always prevails but that thought can occur.

The drop in pressure that Space Ex saw in their data was very short, less than a second, before the overpressurization and rupture of the LOX tank occurred. The best scenario that Space Ex was able to come up with was that the initial "sound" that high speed accelerometer data was used to triangulate to the area of the COPV's seemed to occur at the same time as the drop in pressure. If this was indeed a strut or bolt breaking then the COPV would begin to rapidly rise in the LOX tank, ripping the connecting tubing that attached it to the other COPV's as it moved. Space Ex engineers theorized that the tubing could have kinked and shut off the leaking Ghe for a few milliseconds which would explain the return towards "normal" pressure on the Ghe manifold.

I think we have reached the point described above concerning an exception to the universal laws of physics with this explanation. Or... as a friend I worked with for many years used to explain in a crude way; bull$%^t. He would usually do this quite openly in a feigned sneeze at high volume; bull$%^t as he covered his mouth. In order to understand my incredulous disbelief let me explain a little further.

A water hose can indeed kink in such a manner that you can shut off the flow of water at low pressures. I have even seen brake lines of small diameter kink such that hydraulic fluid can be restricted enough to defeat the balance on a brake system. What I have never seen and will defy anyone to produce is a 5500 PSIG Ghe line that is kinked enough to shut off the leak of Helium. In the configuration that Space Ex uses of their COPV's inside LOX tanks there are several different tanks tied to one tubing manifold. If one tank becomes detached and begins to rapidly rise the line would have to kink in two directions at once for the pressure to return to "normal." It would have to shut off leakage from the rising tank and the tank that it was tubed to at the same time. If one such kink is impossible I don't know how to describe the statistical impossibility that two simultaneous kinks would represent.

There is more....

In the constant effort to save weight Space Ex used titanium tubing to make these manifolds. Titanium is both lighter and stronger than the stainless tubing usually used in such applications so they were able to use extremely thin walled tubing for this assembly. In other words, this tubing has great strength for retaining internal pressure but almost no shear strength to resist tearing apart. If a strut assembly were to break and the COPV to start a rapid ascent in the tank it would immediately tear the tubing apart instantaneously releasing a large volume of helium into a tank with a very small ullage. In other words, the LOX tank would rupture AND there would be no rise in Ghe pressure after the initial drop in pressure.

Add to all of this the fact that all preliminary data suggests that the same type of COPV in the same second stage LOX tank just experienced a "massive breach" that destroyed a second Falcon 9 as it was filled with propellants while sitting on the launch pad and the story gets even harder to believe. Preliminary data also suggests that there was a large drop in pressure followed by a similar rise in pressure in this same Ghe system right before this vehicle exploded.

Is there a scenario that matches this sequence of events? It turns out there is and it isn't counterintuitive at all once you understand the COPV failure mechanism. More on that tomorrow......

Confusing Data and Assumptions about Metallurgy

On September 1, 2016 a Space Ex Falcon 9 exploded on the pad at Kennedy Space Flight Center. The Falcon was in the process of loading propellants when this explosion occurred. Preliminary reports suggest that a large breach occurred in a helium tank in the upper stage which after some .9 seconds caused the rupture of the LOX tank it was contained within. The resultant fire destroyed much of the Falcon 9 and severely damaged the pad itself.

There is some careful wordsmithing going on at the moment to suggest that this accident has no relationship to the earlier Falcon 9 that exploded in flight but the truth of the matter is that the ultimate destruction of both vehicles was caused by the failure of a helium tank inside the upper stage LOX tank. Space Ex, who had earlier determined that the first loss was caused by the failure of a mounting strut that holds the tank in place during flight, immediately suggested that there is no correlation between the two failures as this strut is not under dynamic load during propellant loading. While this is true, it skips over the fact that only Space Ex believes that it fully understands what happened to the first flight. An independent NASA investigation into the same incident suggests that while the strut issue was a problem, there are several other possibilities that could have caused the same incident. In other words, while everyone agrees that a rapid overpressurization of the LOX tank caused the incident, everyone does NOT agree about what caused the failure of the helium tank that led to this overpressurization.

The first Falcon 9 that exploded in mid-flight was most definitely experiencing dynamic loads that are not present during propellant loading on the pad. Therefore, it is extremely unlikely that a strut failure occurred to cause this explosion. However, it is worth backing up a little bit here to explain how Space Ex decided that the strut failure was actually what caused the first accident.

Early data from the first incident presented some seemingly conflicting and contradictory data. Telemetry system data suggested that there was a very brief drop in Helium pressure immediately previous to the explosion. This would make sense if a tank experienced a sudden leak or breach of some kind but there was also data that suggested that the pressure immediately returned to normal before the actual explosion. Engineers from Space Ex and NASA were confused by this information to say the least.

Early on, Space Ex was concerned about the bouyancy effect of the Helium tanks within the LOX tank. Bouyancy in LOX is little different than bouyancy in water and most everyone understands that holding a balloon underwater is problematic. The same thing occurs in a COPV pressurized with helium in a LOX tank. As the G forces increase during launch the bouyancy increases. In other words, the upward pressure on the struts that hold the helium tanks in place increase as the rocket ascends rapidly. Due to the timing of the incident on the Falcon 9 it seemed that this problem occurred simultaneous with a very high G loading on the flight.

The second clue that led investigators to look at these struts was some acoustic data from microphones/accelerometers on the vehicle. The data from these intruments is taken at a high rate of speed that is inherently necessary to gather vibration data for analysis. As I have discussed in here before (see Update Rates) digital data systems take snapshots of pressure, vibration, and temperatures. These snapshots are taken at varying rates dependent on the type of data you are trying to collect. These snapshots are then arranged on a plot and a line is drawn between the points on the chart to create a graph displaying this information vs. time.

One of the problems with digital data is that is can easily be used to draw graphs that do not resemble actual events. For instance, if you take digital data on a repeating sine wave that operates once a second you can accurately represent this sine wave if you take at least ten snapshots per second. However, if you take only two snapshots in this time period you will wind up with a graph that doesn't even resemble a sine wave. It is accurate data at that point but it completely misrepresents a sine wave. In other words, the snapshots are accurate but the resultant graph is bogus.

There are standard formulas for deciding data rates for all manner of instrumentation which I won't go into here, but it is also dependent on both the instrument you are using to gather the data and the speed of the event you are trying to capture. In the case of the Falcon 9 that exploded in flight the data gathered from the Helium pressurization system was probably rather slow in terms of trying to capture the event that actually happened. I don't know this for a certainty as I have not personally seen the data but a pressure trandsducer that is being used to monitor tank pressure there is typically no need to monitor it at a high rate of speed as the pressure is not expected to change extremely rapidly. If you knew you might want to use it to decide exactly how and when something explodes you would run it at an extremely high rate of speed but that is not what this system was designed and built to do.

Ideally, one would set all such systems up for such an eventuality but running at extremely high rates of speed on numerous channels costs money and if you aren't convinced you will ever need this high speed data you simply don't design systems to accomodate it. When we are purposefully taking COPV's to failure we would typically run our data collection pressure channels at 50,000 Hz. In other words, we would take a snapshot 50,000 times a second so that we could see exactly what the pressure was when the tank ruptured. Gathering 50,000 Hz data is not that difficult with today's systems but storing it and being able to analyze it later can be problematic and expensive.

I suspect from the information I have seen released that the pressure system Space Ex was using on the helium tanks was on the order of 10 hz. Again... I don't know this for certain but it would make sense economically and technically as they were not expecting to see rapid pressure changes in this system to begin with. It would also explain the delay between the drop in pressure and the overpressurization or loss of the vehicle. Even if they were running at 100 hz it is still not fast enough to have a lot of data as to what had actually happened in the helium system.

Space Ex was also running accelerometer/microphone data systems at several locations on the vehicle to monitor vibration during the flight. These types of instruments operate at much higher frequencies inherently because they are looking for vibration signals in the thousands of cycles per second. Using this high frequency information they look for vibrational problems that might create positive feedback loops of resonant frequencies that could damage or destroy the vehicle. This is another problem with space flight that is always a concern. You don't want to set up positive feedback loops that destroy your vehicle. Instrumentation looking for these loops is monitored and systems are throttled specifically to avoid these issues.

Using this high speed data, Space Ex determined that there were two significant incidents that occurred at different times. The first was a significant "sound" or detectable vibration and .9 seconds later the vehicle exploded. By triangulating the various signal locations Space Ex determined that the first "sound" came from the area where the helium tanks were located. This led back to their original concern about the bouyancy of the helium tanks in LOX and how it was affected by the G loading during ascent. If the original "sound" was a strut breaking, the rapid rise of the helium tank in the LOX tank could have followed. The resultant collision with a wall of the LOX tank or the top of the tank would have precipitated the instantaneous failure of the COPV, releasing 5500 PSIG of Helium into the LOX tank and overpressurizing it immediately.

Space Ex began testing mounting struts that they had in stock to see if some of them might fail at lower torque ratings than their specification. What they found was that several of them did fail at much lower ratings than their specification. Meallurgy studies found consistency problems at the granular level in these struts. Steel struts, just like steel bolts are rated for shear strength and manufactured accordingly. The second part of this problem has to do with the fact that they were being used to mount helium tanks inside a LOX tank. If a steel strut has minor inconsistencies in the granular structure but it is highly overrated for shear pressure this is not a problem. Space Ex soon came out and said that some of these struts failed at 5 times lower pressures than they were rated for. They also refused to release the name of the manufacturer of these struts but stated that from this point forward they would individually test each strut before use.

All of this sounds reasonable except for there is no mention of the more important fact that typically carbon steel struts are NEVER used in cryogenic applications. Exposing stainless steel to -297 degrees temperature changes its shear rating dramatically. Minor granular inconsistencies become major catostrophic failures under these conditions, which is why you do not use any type of steel struts in such conditions without comprehensive metallurgical pedigrees. Space Ex has been in a constant running battle with its NASA oversight groups from the beginning of its existance because of its unwillingness to do due diligence on configuration control issues such as this one. When this Falcon 9 exploded in mid-flight there were numerous parts flying on it that Space Ex could not identify as to origin or pedigree. This was not limited to struts, nuts, and bolts but went as far as valves, regulators, and all manner of complex components. In other words, they completely lost configuration control in their haste to launch vehicles on many of the systems on their vehicles. This is a much worse problem than a few struts that failed during testing. It is a problem that will have ever more serious implications in the future if it is not straightened out.

Space Ex used the data it had to locate an issue. The struts they were using to mount these tanks had basic flaws in some of them that could have caused this accident and they absolutely needed to fix this issue before continuing to launch vehicles. Unfortunately, there is no proof that this issue is what caused the loss of the Falcon 9 during flight. It makes a plausible story and it was definitely an issue that needed to be rectified. However, neither any of the investigators on Space Ex's team that did not work for Space Ex nor the independent NASA investigation team were convinced that this was THE cause of the loss of the Falcon 9 on June 29, 2015.