When Geeks Kill

This is a total geek post, but remember when I said I write code for a large governmental agency?

WASHINGTON – The chairman of the House Veterans Affairs Committee said Thursday he would investigate software glitches at the VA that put patient safety at risk, calling the problems a sign of a "dangerous lack of accountability."

The Associated Press reported Wednesday that patients at VA health centers were given incorrect doses of drugs, had needed treatments delayed and may have been exposed to other medical errors due to the glitches that showed faulty displays of their electronic health records.

There is no evidence that any patient was harmed. But the glitches, which began in August and lingered until last month, were not disclosed to patients by the VA even though they sometimes involved infusions of drugs such as heparin for up to 11 hours longer than necessary, according to internal documents obtained under the Freedom of Information Act. Heparin, a blood-thinner, can be life-threatening in excessive doses.

The VA's recent glitches involved medical data — vital signs, lab results, active meds — that sometimes popped up under another patient's name on the computer screen. Records also failed to clearly display a doctor's stop order for a treatment, leading to reported cases of unnecessary drug doses.

Documents obtained Thursday by the AP show the glitches may have extended further. In some cases, VA medical centers reported that automated dispensing machines sometimes printed out the wrong patient name when filling prescriptions for outpatients, according to an internal VA memo dated Nov. 5

This is a huge deal, obviously. Coincidentally, I have a new patch that was supposed to go out nationally last Monday. It's related to another patch for the Pharmacy system. That Pharmacy patch requires that my patch be installed first as it provides some database entries required by certain API calls.

Both patches are currently held up in a new-ish patient safety review group. My patch has nothing to do with patient safety issues, but the Pharmacy patch probably does.. and it's obvious what's going on. The problem is, the patches are time sensitive. There's a narrow "window of opportunity" to release the patches, because a different part of the Pharmacy system gets updated on a regular schedule and those updates would be regressed if not accounted for in the new patch. So, when missing the window, the patches need to be updated, and because something was changed, the entire review process (and it's long and involves lots of people) has to be started all over again. The length of time that takes has increased to the point that it appears we'll never be able to make the window. The patches will need to be updated and reviewed over and over because of missing the window.

That is a major problem.

In the system, a patient is identified internally by a unique number akin to a SSN. When a patient is selected, that number is read in, and other variables are populated by reading various fields off the database based on that key identifier.

Suppose you wrote a new functionality, and part of that functionality is selecting the patient. Suppose you forgot to read in the patient name based on that patient's key. It's quite possible that the variable for the name contains the name of the last patient that was selected in a completely different part of the system. That's bad, because when the patient name is displayed, it's wrong.

Coding these systems is so complex it's rather easy to make errors in the way that you set variables. It's not object oriented, so it's not like you have one class that you reference from anywhere to get a standard set of data, like complete patient data. You have to pick it all out of the database as you go.

I'm not the quickest coder in the world by far. I've seen some bugs where it's obvious it could never have worked in the first place. Suppose you have some condition where one of two things could happen. The first condition happens the vast majority of time, and it works. I'll see the bug in the second condition and it's obvious that it never would work, so you know the coder didn't test his condition for both branches. He just assumed it would work correctly. I don't do that. For every singe IF in the code, I test that the results that are expected is what actually happens, but even that isn't fool proof.

Technically speaking, that's not the fault of the coder. The SQA folks are supposed to have a script to test with that details every single possibility. They can be massive and complex, and there are often disconnects between what a coder has in his head and what SQA does for scripts. In this case, nobody saw the problem with the patient identifications. It looks like there are two separate areas where variables were not cleaned up after usage, and not set correctly on entry.. that would be the patient name and medication dosage. Not seeing physician orders is something different and very strange.

This area of the system is the highest sensitivity in terms of patient safety. There has always been a great deal care that goes into it, but ultimately software development always comes down to human beings, and people make mistakes. The solution in a government bureaucracy is always to throw more bureaucracy at it, slowing things down, but never really changing anything. It's taking longer and longer to release software.

I used to specialize in the Surgery software, which is also highly sensitive. Surgeons and nurses use it in the OR, and to track everything that goes on. Suppose I tweaked something with the functionality of the procedure information, and neglected to set variables correctly. You might need an appendectomy and get a kidney transplant instead!

In other words, you could kill somebody with bad data in Pharmacy and Surgery. The people that worked on those Pharmacy problems know every well that it was their code that caused the problems, and I'm sure they are highly relieved nobody got hurt while they fixed it. I can't even begin to imagine what it would feel like if something I wrote hurt somebody, or worse.

It's just that the development is so complex, it's hard to explain. I've had to fix bugs in software that I wrote and was subsequently released and installed in hundreds of hospitals nation wide. I just looked at a patch I wrote in 2006, to fix bugs in a patch that I spent a year and a half on. There's a total of 13 bug fixes in it. Many are related to bad screens of user input. For example, valid user responses are variable, possibly say 1 or 2 or 3, and a user enters ($$*, the system wigs out and crashes. That's not great, but at least the system halts on bad data and doesn't just accept it and move on. Depending on how the coding is done, it might take the bad data and try and use it. Yes, there are built in API's for user input that screens for bad data, but sometimes complex conditions don't let you use them.

It's incredible the way things are inter-related. I suppose one could consider the API's in the system as faux object oriented programming. If you don't use them correctly, it's not going to prevent bugs.

The bottom line is, coding in medical systems is super complex and very nerve wracking. It takes forever to go from concept to completion. The quality of the software will always reflect the ability of everyone involved in it's development, and situations like this current one at the VA causes huge heartburn. What was slow and methodical before becomes paralysis when you're worried about a Congressman calling you.

And you want to know what's even more nerve wracking? Hiring an outside contractor to write software outside the system that will interface with data inside the system. If you ever hear the term "authoritative source", be afraid.. be very afraid. Think of it as local caching, but with your medical data. Yow! And to make you even more afraid. The VA has a better safety record than the private sector health care. Think about it.

Rumor is, our new Prez is going to dump 50B (that's a capital B) into VA IT. I'm still not sure if I think sucking the VA into a single payer insurance system would be a good idea, or always keep it seperate. Vet's do have unique needs.