Engineering Excellence

It's been a long time since I've written one of those really long Dilbert-like posts that explain how things work, how they fail, and why political ideology affects everything, not just politics.

This will explain why patients at VA hospitals get the wrong dossage of medications, and why space shuttles explode, killing astraunauts.

This will also explain why Microsoft will eventually rule the world... even though an argument could be made that they already do.

Yesterday, President Obama detailed plans for the Veterans Administration to change the way they keep medical records for millions of veterans, making them more portable across VA facilities, and even inter-operable with the Department of Defense (DoD).

I'm going to explain how it all works, how the technology was developed, and why doing what Obama has proposed is going to be a monumentally difficult thing to do.

I have to be sort of careful about what I say about these things, even though I'm mostly identity anonymous here. The VA is managed by people who think they should manage like private sector company managers operate. That is, they issue gag orders rather than embrace transparancy.

I am not kidding.. it may seem counter-intuitive, but the VA actually gags everyone associated with anything that goes wrong. It's not unexpected that... say.. a car company would issue a gag to employees when a car demonstrates the propensity to burst into flames when getting in an accident, but the VA is a government agency that should be completely transparent to the people who own it.. that is, every single American as tax payers.

The VA does not work that way however. They will hide what they can hide. Every governmental agency does that, NASA, the EPA, FDA, etc. This is because, again, these agencies are managed by people who think they are managing private sector companies.

For example, the last 7 years of so, the VA has been trying to integrate COTS (commercial off-the-shell) products into the home-built software platform that is used. One very large project to replace the financial system of the VA was called CoreFLS. That project failed spectatularly.. many millions of dollars were wasted.. and news paper coverage followed, exposing the waste of tax payer money. It was a collosal failure where the VA paid an outside vendor millions and millions of dollars, and what the vendor developed was a failure and the entire project scrapped.

The VA gagged employees and contractors.

I worked on a project that was initially called "Billing Awareness", and was later called "CIDC", an acronym for "Clinical Indicator, Data Capture". The word "Billing" chaffed a lot of people, for the same exact reason why Obama recently scrapped plans to bill private insurance for service-connected patient treatment. The word "billing" sounds like the vet would be billed, but rather than explaining that the vet is never billed regardless, it was determined that it would just be easier to change the name of the project.

The CIDC project was designed to integrate a COTS product with the VA system in order to make it easier to bill private insurance for veteran treatment when appropriate. When the project failed, we were all gagged.. and in the government, a gag order consists of notice that if we are contacted by any journalists, we are not to speak to them but rather direct them to VA managmenent.

In my view, that is extremely unethical.

To explain why things like these projects keep failing, the history going back to the very begining needs to be understood, and some terms clarified.

What is engineering?

Engineering is the intellectual activities that result in the creation of a fundamental part, a thing that does something.

No offense to my networking guru readers, but there is no such thing as a "network engineer". There are "network administrators". These people are like super users of networking equipment, such as routers and switches and such. The actual engineers are the people that designed the innards of the parts.

The engineering is the hard part.. and the most important part of anything that happens anywhere along the chain of an organizaiton that does something.

The terminology is important.

......

VA & DoD IT history;

Both the VA and DoD software system were built in-house by federal employees begining in the early 1980's. The language it's built in is called MUMPS or more simply, M, which was developed in the late 1960's. It's an interesting language as it's both a quasi-operating system for the host computer, as well as a language for software development. The language sits on top of the UNIX or VMS operating system, but all data "files" associated with the lanauge are all built in to the language environment.

There are only 16 commands in the entire language. This is different than a lot of new languages that have hundreds, if not thousands of commands.. but in reality, the commands in new languages are actually just pre-built API's that the language vendor has provided.

Because the M language is fairly "low level" in it's structure, there are limitless ways something can be done. The software engineers at the VA (very very smart people) designed an entire database structure and API suite that developers could use to create applications in a standard way. It is simply amazing.. It's very analogous to the government employees developing something similar to Microsoft Windows for use in the agency. To be sure, it's not a graphically based system like Windows is, but the lower level technology is similar.

The system is, ironically, called "VistA". The low level database structures and API suites are called "kernal". Developers at the VA do not need to know exactly how kernal functions, but it probably takes the better part of a year to learn to write software within it. There are developers on the kernal team who continually revise the platform though, and most application developers at the VA have a very detailed understanding of what kernal is doing. I've recently written some software for kernal. It's actually some modifications of kernal software that was originally written by a Russian guy, and it's weird that software can actually be stylistically different when written by people from different cultures.

When kernal was developed in the 80's, there wasn't a large IT structure at the VA. It was all driven by the engineers, and I cannot overstate how awesome it is as an engineering achievement. It is still used to this day in the VA and DoD, and many other companies have adopted it and are actively using it. I don't know of any other software platform that has lasted this long.

So.. we have this platform that looks like;

native operating system (Unix) --> MUMPS language --> VA Kernal --> VistA --> package

Actual applications needed to be developed that people can use. In the VA, different applications are called "packages". These are discreet domains that abide by the built in rules of the kernal. The packages generally follow the departments of the VA. For example, there's an enrollment system that tracks patient demographics, a pharmacy system, a laboratory system, a surgery system (which I did a lot of work in), a financial system.. and so on.. it's huge huge huge.

These different packages can interact with each other by providing their own API's to access data, and the developers of the package agree to make sure to keep those API's up to date. Historically in the VA, each package had it's own team of developers because they are very complex. It takes about a year to learn kernal, probably another year to learn a package before somebody can be very highly qualified to write software in that package.

Each VA facility (hospital) has it's own computer with VistA running on it. They have their own data for their own patients, and so, when you consider all these things I've described.. when the President says he wants a single electronic medical record for a vet that can follow that vet from one facility to another, you can see how complex that is. The vet's data is typically spread throughout many different applications at one particular VA facility, and now all of that data needs to be packaged up and available from anywhere... including integration with the vet's active duty records at the DoD.

And to complicate it further, there are developers at individual facilities who have made local modifications to their own VistA systems that are not part of the national standard software suite.

And to complicate it further, there are often no particular standards for the data that is contained within the database. For example, a yes/no question for something may have "N" and "Y" in one part of the system, and "NO" and "YES" in another.

So.. the VA developed a system called the Health Data Repository, which is located in Austin Texas.. which is designed as a central database for all veteran data. It is a mystery to me how it actually works, and what data it actually collects. I've not had any experience in what is does, or how far it can be leveraged to achieve the President's goal.

One of the problems with the massive amounts of data at hundreds of facilities is that the terminology used is not always consistent. For example, the ingredients for a particular medication may vary slightly from facility to facility. The VA has a "package" that is designed to standardize everything within VistA. I'm currently working in that package. It's highly complex.

To complicate things further, the powers that be at the VA want to retire VistA and replace it with a COTS systems... you know.. the same type of thing that has been failing and failing at the VA.

VistA is so massive with so many packages that it is impossible to pull the plug on the entire thing, and bring up a completely separate and new system instantly. You can't have downtime on the system because you treat patients 24/7. What the VA is trying to do, is unplug one package at a time and bring up a COTS replacement for that particular package... one at a time. They're schedule for final completion keeps getting pushed back. Not really surprising.

I'm working within the terminology standards package to help the VA replace the entire pharmacy system with a COTS system that another company is developing according to requirements and standards provided by the VA.

Okay.. does that all make sense?

You have a pharmacy system in the VA's VistA system that is tightly integrated with the rest of the VistA system, and they want a contractor to make something that they can just "plug in" and go with zero interruption in service.

That failed in CoreFLS and failed in CIDC, costing tax payers many, many millions of dollars.

In CIDC, the COTS billing system was going to collect data and maintain it's own database of patient data that would be used to bill 3rd party insurance. That same data would be collected from many packages within VistA. I think it was a total of 9 packages that were affected.

In VistA, the data from the various packages was to be collected in one central package called PCE (for patient care encounter). That data would be fed to the COTS system, which would have it's own database.

If you know anything about software engineering and database administration, that should terrify you. Sit in on a meeting and hear the word "authoritative database", and your blood runs cold.

Again, VistA is a system of many packages, each with their own database but it all exists within the same framework of kernal and the M language. You can relate those databases by pointing them at each other, and VistA is designed to support that. It's high quality data integrity..

Take that and add a COTS system outside of VistA with a database that is not even located on the same computer, and you have to try and "synchronize" things. That introduces a great number of variables that cannot all be accounted for and your assurance of data integrity turns to nil.

If you're talking about, say.. Google caching, that's not a big deal because the data that is replicated across separate databases is simply links used in a search engine. No big deal if something comes out of sync... but when you're talking about that method as a standard within a system built on PATIENT data.. that is how you kill people. I'm so.. not... kidding.

Sure.. in CIDC, it is just billing data, but it's a paradigm shift in the VA and if you go and try and unplug clinical systems (like surgery) and plug in a COTS system, all of a sudden, you'd rather just go to a county hospital for treatment instead. Know what I mean?

.......

Change in VA IT culture:

The VistA system was built by federal employees of the VA. It is awesome.. the engineers were awesome.. it is so freakin' amazing and beautiful.

The VA later partnered with an outside company to provide software engineering talent. The term "partnership" is important. This wasn't just an outside company providing contract services, or building COTS products. This was a true partnership, where you really couldn't tell the difference between an engineer that worked for the outside company, and a VA employee. Everyone worked by the same standards at the VA. Everyone worked in tight knit groups, knew their peers, and the transition was relatively smooth.

A few years back, the VA again changed the way IT resources are allocated, by dissolving the partnership and employing contractors at many companies. You can imagine what happened. Not good.

....

The IT food chain:

I've noticed an evolution in the way the IT culture functions in a broad sense. I've been in this business for quite a while, and the change is not very positive, but is not surprising.

When you engineer a product, the most important people involved in that effort are the engineers. That is where the talent is. That is where the smartest people are. That is who should be paid the most, and who should be consulted in every decision. That's how it used to work. That is not how it works now.

In many companies, and certainly in many federal agencies, the least important people and the least consulted are the engineers. This is because the management are not engineers, and they simply are not smart enough to understand how things function, how they are built, and they do not understand the crucial details that determine whether a project is successful or not. Management is more focused on budgets, politics, and trivial activities such as "project planning". That is all meaningless crap that destroys projects. It is what has killed all these projects at the VA, and it will not change.

This is also the same paradigm at NASA, and is what causes shuttles to explode.

During the CIDC project, I was rarely consulted about requirements for the COTS billing platform. Managers at the VA talked to managers at the software vendor. Engineers at the VA did not talk to engineers of the COTS system.

The "go-betweens" used now are typically the "functional analysts". They are sometimes called "business analysts". The function of these people is to evaluate the "requirements" of the software design against what the "business rules" are, and draft documentation that tells the software developer what to do.

The theory behind this is that it frees up the developer from having to spend time doing things other than writing software. The introduction of this function in the software development life cycle is a huge mistake. It causes projects to fail.

In the VA, at least, the functional analysts are "subject matter experts". It is good to have people who know how things functionally work. For example, if you are designing changes to a pharmacy system, it's good to have somebody available to tell you if it would cause a problem, or help, or meet a need, etc. It is highly damaging to have those individuals do the actual designs.

Frankly, they are not nearly as talented in the software world as the developers. However, in the VA and many other businesses, the analysts are placed above the engineers in the food chain.

Again, this causes shuttles to explode and patients to get the wrong medication dosages.

Every so often, a software engineer will go into a management role, and they can be quite effective, but it is rare that happens. Engineers typically don't want to deal with all the BS of management. In the VA, the manager of the surgery package is the guy who wrote much of the package in the first place. He's probably the best package manager in the VA. He knows how things work, knows how to move projects forward, and gets things done. Pretty much everyone else is a failure to one degree or another because the food chain is upside down.

This is why Microsoft will eventually rule the world. Their top management are engineers, and they value the talent of the engineers who actually make things.

Microsoft has made some mistakes, and you can surely question their business practices, but their operating systems get better and better. Windows 7 is going to be a huge success. This is because they recruit the best engineers, and they pay them well. The engineer is at the top of the food chain, not the bottom.

When you consider it in terms of political ideology, Conservatives value CEO's and their massive pay. Liberals value the talent (in this case, the engineers), and understand the CEO's are irrelevant to the extent that all the management can do is get in the way of the engineers and ruin projects.

.....

The future of VA IT:

If the VA wants to get anything done, they need to change their contractor model, and rethink their whole "re-engineering" model. It's been a failure, and it will ultimately be a monumental failure. I don't have much hope for it because the VA simply shifts people around and there is no real accountability for anything.

VistA is Godzilla. It's a huge monster.. but it's also beautiful and it works. It's just old, and people don't like old IT. Managers feel like they have to earn their pay, and to earn, they have to do something.. and doing something means.. replacing VistA with a new, shiny system that looks like modern software.

I don't understand the thinking.. but all managers feel like they have to change something, even if it's working. Because of that, they've actually wasted hundreds of millions of dollars. There is ZERO question that the VA has flushed it down to the toilet, and there is ZERO question who is responsible.. but just like the banking collapse, they take ZERO responsibility for what they've done.

Congressmen get upset.. and they hold hearings.. and people get shuffled around.. and nothing changes for the better. They keep trying to destroy VistA, but instead, they just destroy money.. the same is if they took big piles of it, doused it in gasoline, and threw a match on it.

Meanwhile.. all the talent at the VA that built VistA in the first place is retiring, or long gone.. washing their hands of the whole thing, after they predicted all of this would happen.

There is a bulletin board system at the VA, and I remember when the announcement went out that VistA was going to be "re-engineered". The BBS exploded with warnings from the engineers of the disaster that was to come. It came.. big money wasted.. gag orders issued.. zero accountability.. because nobody actually did what the engineers recommended. Instead, the big money employees.. the GS 15's.. the smart managers with MBA's just knew they were the designers of a new VA. They were the "leaders" that would bring about a new VA.

They were wrong.. they are wrong.. nothing has changed, but a degradation of the VA IT environment and standards.. and all that's happening now is that they are firing bazzokas at Godzilla, but you can't kill Godzilla. Everyone knows that. There's no way. You can live with Godzilla, and make it even better.. and you can put a glossy sheen on it so people think it's new.. but the underlying architecture can be as it's always been.. and it'll work for many more decades..

And when you see articles in the news about patients being put at risk.. this is why... and the VA will respond by deflecting and promising to change and improve things.

I can tell you how to fix it.. every long-time engineer at the VA can tell you how to do it. They don't want to hear that answer. They won't hear it... because they went to Harvard.. they have an MBA.. they got a political appointment because they know somebody who knows somebody. They're convinced they're smarter than the people who made Godzilla in the first place.

In the end, though, everyone forgets that the most important thing is the Vet...