Killing Superman: The Importance of Good Laboratory Practice

People who are not laboratorians often don't understand the importance of Good Laboratory Practice. And yes, I meant to capitalize each of those words. Whenever you hear a scientist talk about Good Laboratory Practice (GLP) you should hear those capital letters because GLP is not the good practice of measuring twice and cutting once, it's the good practice of paying your taxes.

Good Laboratory Practice standards were first discussed in the 1970s when the realization came to light that you can't always take scientific data at face value. Before that time, the assumption was that the study was always done properly and recorded precisely. Pretty naive when you think about it. That's not to say that any action outside of GLP standards is malicious, all too often it's done out of ignorance, which is why it's so important to have well qualified and fully trained people who understand the importance of GLP.

People who don't understand the importance may balk at GLP because it is detailed, it is rigid and there is very little wiggle room. What does the minutiae matter? Isn't it just the final outcome that matters? Why the obsession with the process?

Killing Superman: A Parable on Good Laboratory Practice

One day in Gotham an analyst at Gotham Laboratories Inc was preparing her Hal 9000 to analyze water samples for Kryptonite levels. Lex Luther was at it again and there were a group of wide eyed orphans clutching teddy bears, stranded on an island surrounded by rising water. It was a sad sight, and Superman was chomping at the bit to go rescue the tiny tots.

There was just one problem. The water surrounding the area was not only fast moving and too dangerous for the mere mortals to rescue the orphans, but it was known to often be contaminated with Kryptonite. It was too dangerous for Superman to plunge in, he needed the laboratory tests first.

The Hal 9000 was calibrated, but when the analyst looked at her low Kryptonite control she could just barely see it. This wasn't good. The standard was at 5ppb. Although a low environmental level, it was the highest level of Kryptonite Superman could function with. Recovery was supposed to be at least 80% so that they could be sure if there was Kryptonite there, they would see it and thus be able to warn Superman. Not seeing the control was just not going to cut it.

Perhaps the standard was old? So the analyst opened a new bottle and ran it again. Nope, no good. Perhaps the calibration standards needed to be re-prepped? So the analyst re-prepped the standards. Nope, still couldn't see the low Kryptonite standard properly.

The analyst had worked at Gotham Laboratories Inc. for a long time so she knew the Standard Operating Procedure and she had done the first two steps for correcting a low control. The next two instructions in the SOP were to troubleshoot the instrument as needed, and then if that didn't work to contact the manufacturer. No problem, troubleshoot as needed. The instructions weren't anymore direct than that because, well... how do you document every possible thing that could go wrong and it's fix? I mean, that's why you have highly trained individuals in those positions.

But alas, Gotham hadn't quite embraced the GLP standards yet and where in another laboratory only the senior level analysts would do the troubleshooting, that wasn't a consideration here. I mean, that's what the checks were for on the instrument. If they didn't pass then you knew it wasn't working right and if it did, then it was. The outcome was simple.

The analyst knew there were things she could probably do like changing the lines or cleaning the cones or giving the Hal 9000 a good pep talk but the water was rising, her boss was being a jerk, the finals of American Idol were on TV and well, the way the Hal 9000 kept calling her "Dave" was creeping her out.

The control just wasn't visible so the answer was simple. She'd just add more. So taking her same low level Kryptonite standard she simply doubled the amount of her aliquot. Restarting her run and ignoring the question of "What are you doing Dave?" she watched for the control's results. It was beautiful! 90% recovery. Now all she had to do was run the water samples, Superman could save the orphans and she could go home.

I mean, after all, she fixed it right? The control passed. Well, not exactly.  She did indeed get her run to pass, but she didn't fix the problem. Whatever was making the Hal 9000 unable to see the low level of Kryptonite was still there but the run passed. Now of course the problem was that the point of the test was not to get the control to pass. The point of the test was to get accurate and reliable results in those unknown water samples. Running the control was just a way to be sure the instrument was working properly.

Those water samples that had been taken so carefully across the gradient of water Superman would have to pass were analyzed and things looked good. They were all below the low level control. Superman would be safe! Hurray!

Her boss was happy, the people of Gotham were happy and Superman went into the dangerous passage to get those poor orphans off that cursed island. Now the roof above the treacherous water was spiked and dangerous so Superman would have to swim, but he knew he could do it safely because the run passed and the Hal 9000 didn't see any Kryptonite in the water.

As Superman swum, he started feeling a little ill and half way there he went under. The crowd was stunned! What could do this to Superman? Only Kryptonite surely and they had tested for that. The run had passed!

Oh, but alas, double spiking the standard hadn't fixed the instrument, it only masked the problem and while it wasn't written anywhere that she couldn't do that; if the analyst had been familiar with GLP she would have known that changing the procedure to make a test pass was wrong and that forcing QC to pass can lead to erroneous results.

But alas, Gotham had no GLP and Superman was dead. Those poor orphans.

And thus the importance of Good Laboratory Practice; because the point is not to get the controls to pass, the point is to get accurate results on those unknown samples.

State Experts vs. Defense Expert: Us versus Them

The most heated seminar I have ever attended was one at an annual meeting of NEAFS. (The Northeastern Association of Forensic Scientists for those not in the know) The seminar was titled "Debating the science in Forensic Science" and featured a panel of both state and defense experts as well as a few college instructors. This is clearly a recipe for high tempers!

One person on the panel was an academic...a very outspoken academic...who had not actually performed the work she was passing judgment on. As one can imagine, this was not well received by those in the audience. The effect of her speech (whether she intended it that way or not) was that crime lab staff were biased towards the state and that their work was not held to the same scientific level that research is held to. I would argue that neither of those statements are necessarily true.

Over the years of going to conferences I have found that there are two real constants in the forensic community. One is the passion involved in the work. No one enters forensics for the money! Every single one of them is passionate about what they do. The other constant is the duality of us versus them. "Us" being state, federal, or municipal employees and "them" being (dramatic music) defense experts.

The unfortunate fact is that there are defense experts who will simply testify to anything for the paycheck. Their science changes based on the case and what is convenient to say at the time. They lack credibility and no matter how credible the other side, there is a taint to having a "battle of the experts" with someone who will say anything for the right amount of money.

The role of an expert witness is simply to explain the science as objectively as possible in terms that a lay person would be familiar with. It's very simple. We don't need to make judgments on guilt, we don't need to argue legalities. All we have to say is the absolute truth to the best of our knowledge and explain it so that those without a science background they can still understand. Easy.

Now, have I felt a person was guilty during a case? Of course. I'm human. Did that change what I would testify about? Absolutely not. In fact when the scenario presented by the defense was supported by the math I made sure that the state's attorneys knew it. The attorneys get to argue their version of truth and the fact finder decides. An expert witness should never change their testimony based on whether they feel the person is guilty or not. As long as you always remember that, testifying is easy, and there's no need to worry about statements against your credibility.

There have certainly been cases of misconduct in crime labs, whether through ignorance or intent. These always make the news, and the panel at this particular seminar did discuss some recent instances that had been reported. The academic previously mentioned, extrapolated that to say all crime lab employees are biased towards the prosecution because of their associations with police officers or through overt pressure from superiors to alter a case. This resulted, as you can imagine, in a room filled with red faced people clenching their fists, while the vast majority of the rest of the room frantically raised hands to give their two cents.

I believe overt pressure is truly rare. It is the very rare case where personnel are told to alter results, and I believe that it is truly rare for a forensic scientist to falsify results. Those are the extreme and rare cases.

I do believe however that at times there is a covert pressure. Not to lie or falsify data or anything so dramatic, but in perhaps in not raising as many issues as we may have noticed. We spend the vast majority of our waking hours at our place of employment. People always want that place to be a comfortable place. When you raise issues or say unpopular things, that work place can become a very tense place. Unhappy attorneys tend to be vocal. (This indeed may be an understatement.) So whether we raise issues we know we should or not becomes a difficult question sometimes. I believe that is where most of pressure, if there is any at all arises.

Now as someone who has transitioned from an us to a them I find myself wanting to redefine the term "defense expert" in the forensic community. Being independent does not have to mean being unethical. In fact my credibility arises from the fact that I do have a strong ethical background. My role now is the same as it ever was as a state chemist, to translate the science into layman's terms, to do the math and let the attorneys argue what is "true."

The crime of the academic on the panel was taking a small number of cases where there had been misconduct by the laboratory personnel and applying a bias to all crime lab personnel. The vast majority of forensic scientist are however very honest and ethical in their work. This extrapolation from few to many is the same reason that "defense expert" is a dirty word in the forensic community.

Although we can not force others to behave ethically, we as scientist should always remember to hold ourselves to a high ethical standard. Arguing more passionately for the state does not counteract an unethical defense expert, but in fact adds a sense of bias to the fact finders that simply should not be there in either witness.

As for the panel at the conference, it had to be cut short...