Another B787 Fire?

[conehead]

What do they mean by "stale" data?

[Don Hudson]

Hi conehead...from the Report, Section 1.3, pg. 13:

The EAFR has an integral flight data acquisition function, which receives data from various sources and then transmits those data to the FDR function according to a predetermined schedule for storage into crash-protected memory. If no new value has been received since the last time that a parameter’s value was sent to the FDR function, the flight data acquisition function continues to transmit the last value received from the source. These data are referred to as “stale data.”

For some parameters, the flight data acquisition function has multiple prioritized data sources from which it receives parameter values. For these parameters, a separate source index parameter that indicates the source being used is generated by the flight data acquisition function and then recorded. For parameters with a source index, stale data are indicated by the source index being set to “no source available.” Parameters that have only a single source do not have a source index parameter and thus do not have a recorded indication that the data could be stale. This recording methodology can lead to cases in which apparently valid data continue to be recorded after a parameter source stops providing valid data. This problem delayed the NTSB’s complete understanding of the recorded data during the initial stages of this investigation. Section 2.6.1 discusses issues associated with stale data in analyzing accidents, incidents, and maintenance events.

Also, FlightGlobal, http://www.flightglobal.com/news/articles/ntsb-details-issues-with-787-flight-and-data-recorder-406665/, has an explanation.

Essentially, the recorders were receiving "stale-dated" information from sensors that had stopped reporting but which the EAFR was still transmitting last-received data to the recorders. The problem is partly not knowing which parameter was stale and which was authentic.

The Ars Technica* article captures the essence of the story more succinctly, (bluntly) than the NTSB report including comments regarding Boeing's outsourcing of much of the building of the airplane resulting in non-audited performance by Yuasa and other contractors.

*The Ars Technica Ethos

Ars longa, vita brevis, occasio praeceps, experimentum periculosum, iudicium difficile.

—Hippocrates

When Hippocrates said that "life is short, art is long," he did not mean that art outlives the artist. The "father of medicine" instead diagnosed a basic fact of life: true art or skill takes a lifetime of effort to perfect, and the path is fraught with "occasional crises, perilous experiences, and difficult judgments." Technology is the "art" at the forefront of our changing world, and we're here to chronicle that story and even help with the difficult judgments.

At Ars Technica—the name is Latin-derived for the "art of technology"—we specialize in news and reviews, analysis of technology trends, and expert advice on topics ranging from the most fundamental aspects of technology to the many ways technology is helping us discover our world. We work for the reader who not only needs to keep up on technology, but is passionate about it.

[conehead]

Thanks Don. I think that's a poor design.

[Don Hudson]

I think so too. The CVR was unreadable and the flight data was unreliable. In this technological day and age on a modern aircraft it doesn't say much for the design of the system;- the NTSB sure isn't impressed. Likely it will get fixed quickly but the introduction of the airplane is unfortunately a reflection on Boeing. Wonder what the organizational reasons are for such a massive shift in quality control, auditing of processes, testing and oversight of sub-contractors?

[DEFCON]

"Wonder what the organizational reasons are for such a massive shift in quality control, auditing of processes, testing and oversight of sub-contractors?"

Profit and enhancing shareholder returns perhaps?

[J.O.]

I've heard some fairly strong comments about the current senior management team at Boeing. It's fair to say they weren't complimentary and they didn't come from employees either.

[Don Hudson]

I have too, J.O. - on a number of fronts. There's certainly a story behind these kinds of failures and problems - Boeing builds a great airplane - still does I believe, so something's changed over the past decade or so. I had posted something a while back about the McDonnell-Douglas senior management not being airliner-oriented but production oriented for the military. Outsourcing without sufficient oversight, lack of frequent audits, reduced quality controls and doing things over great distances "far away from HQ" are the usual, familiar reasons behind these kinds of things. Old wine, new bottles.

[boestar]

EVer since they moved the head shed from Seattle to Chicago, things have not been the same.

[conehead]

Well, I believe a lot of the former McDonnell-Douglas management is now Boeing management.

[Don Hudson]

conehead, "Well, I believe a lot of the former McDonnell-Douglas management is now Boeing management.",

Yes I agree - I wasn't very clear in what I wrote above but that's it exactly and I've seen that in print.

That said, other than what I have read in industry publications I have no studies or reports to confirm it but I suspect the McD-D executives gradually changed the Boeing ethos, the way of doing things, the long-term corporate culture from the top down. I believe they've tried to keep the lid on it their labour troubles which may not have been entirely industrial but deeper in terms of disagreement in the way "the ship" was turned after McD-D took over. I'll be watching for a close examination of this in the next year or so - it's certainly worth a book, (perhaps by Diane Vaughan or William Starbuck/Moshe Farjoun).

This is entirely speculative, but I liken this to the intuitive, (and in employees' experience, actual - ref. Roger Boisjoly's emails regarding the O-ring design), understanding some had of NASA prior to the Challenger accident; the "cause" was an 0-ring failure but the real causes were cultural, not technical, and it took them two losses of the Shuttle before they really understood their problem. I think that is the case here; I think if one began walking around the floor in Seattle, Chicago and at Yuasa and other suppliers, scratching the surface a bit one would find similarities. In my view, the regulator, (FAA) relied too heavily for information and engineering assurances on the very group that required auditing and oversight - the manufacturer.

With two thermal-runway battery fires in the fuselage, (in only 52,000hrs of flight time, when probabilities of failure in the order of 10^-9 are required for safety-critical systems), but for good fortune, (both airplanes on the ground), Boeing came very close to the same outcome.

The grounding of the B787 was the first time an airliner had been grounded since the DC10 in 1979...

Some of this is obviously hindsight, but that only excludes some of the assessments of what went on. If these organizational factors were at work and this outcome went true to form, then I strongly suspect that there were many employees within Boeing, within the contracted manufacturers and probably within the FAA as well, who at the time were concerned but were not heeded or heard. Typically, it's a variation on the psychological truism "I'll see it when I believe it" where powerful, large interests with great weight and therefore momentum are at work. Until that realization occurs it is extremely difficult to alter corporate cultural world views without long-term effort or a singular, extreme event such as has occurred here.

What remains puzzling and even disturbing is that these factors which compromise good processes by normalizing the deviances from established, safe practices are well understood. Why would two major organizations, (FAA, Boeing), not be aware and insitute the necessary limitations and boundaries between accepted and unacceptable performance? I can only think of one reason and that is the presence of an entire new generation of senior managers and even engineers who never heard of Challenger first-hand and who have never read the books on the shuttle accidents. Without such knowledge and comprehension and believing that they are doing exactly the right thing, (but without knowing why they think this way), they may have gone down the very same path as NASA did in the early 80's.

Like NASA and the Challenger & Columbia accidents, the initial assumptions were about financial pressures and the willingness to cut corners to make a profit. Vaughan's book shows that these were not the processes at work at NASA. In fact, the very opposite was occurring in that most thought they were doing exactly the right thing, which is what the normalization of deviance does! Even as financial pressures will have been factors, I think this is less about production pressures and the willingness to cut corners. I think this is more about performance pressures. We'll see, as this is studied in-depth, as it certainly should be.

[Super 80]

I have too, J.O. - on a number of fronts. There's certainly a story behind these kinds of failures and problems - Boeing builds a great airplane - still does I believe, so something's changed over the past decade or so. I had posted something a while back about the McDonnell-Douglas senior management not being airliner-oriented but production oriented for the military. Outsourcing without sufficient oversight, lack of frequent audits, reduced quality controls and doing things over great distances "far away from HQ" are the usual, familiar reasons behind these kinds of things. Old wine, new bottles.

I have heard about a dozen variations on the same story from within Boeing, The McDonnell Douglas guys who turned up were infatuated with Airbus and going forward they wanted to build airplanes the “Airbus Way”. This mentality melded with grizzled Boeing executives who coming off the 1995 strike wanted to break the machinists union.

But the lessons of Airbus were applied selectively and things were ignored such as that all the constituent parts of Airbus had independently built and delivered large modern aircraft and the Airbus workforce was hardly an example of unskilled non-union dudes banging prefab parts together. With each pass their interpretation of the Airbus model was further and further dumbed down until it was just a crude interpretation of the supply chain. The rest is history.

[Don Hudson]

Super 80;

It's one variation I hadn't heard, but still it's another very old lesson, no?: "Stick with what you know and do best." To me, the "military fighter approach" vs. the "civilian airliner" approach to design and manufacture do make a lot of sense. I can see the attractiveness of off-shoring but won't pursue a critique of that here.

Airbus' ways have clearly been successful as were the Boeing's ways even though they were vastly different. Airbus has always been a sort of "consortium" - it is their history, and the airplanes they build come from all over the world. I can't speak for Boeing's methods even though I believe they too successfully had overseas contractors - in fact the B787 fuselage structure is a prime example of such success, (though the wing-body joins initially were not...). It is uniform application of the process, and not the process itself that I think is the focus at least for the NTSB.

The sad thing is loss of reputation and credibility, which lasts much longer than the fixes for the original design, manufacture and oversight problems.