I predicted in December 2012 that the exchanges would not be up and running on time with minimal knowledge of how the contracts and budgeting were being run, because the administration was being pretty closed-mouthed about those things. Was I prophetic? Hardly. I just didn’t see how the administration could make things work in the allotted time frame. The development cycle was just too aggressive, even with what my boss used to call a “Shake and Bake” system (take something out of the box, add a few of your own ingredients and roll it out). I thought about the software-implementation projects I’d worked on (not in development but on the server side) back in the days when I was an IT consultant. This seemed a lot faster than anything that any company I’d ever worked with would commit to, even if it had already designed some of the underlying architecture.
Anyone who has ever worked on or, worse, bought a big software application – and this is one of the biggest in history — could have told HHS that the final result would be buggy, late, unsatisfying to users, unable to live up to its billing, and most of all, resistant to upgrades, much less wholesale changes. In the real world, you can’t just order “Make it so!”
I, too, have some experience with software development. This sounds to me like a case of suits saying one thing and geeks saying another. The suits are talking about “glitches” and “too much demand.” The geeks are making it sound like the system might be FUBAR.
1. I have been reading that the process of interacting with databases in order to deal with user identity and security issues is cumbersome, and that is causing the problems, even at small scale. If so, then this cannot be solved by adding servers or by patching a few lines of code. It may be closer to “we need to rip the guts out of the segment of the system that manages data transfer and re-architect it.”
2. My instinct at this point would be to try to separate the functions involved with sign-up from the functions involved with “tire-kicking.” I would think that it would be easier to set up a “tire-kicking” site where people can find out what the plans are and approximately what they will cost given their personal situations. However, because you could not actually sign up, the system would not store personal information or access so much data. Meanwhile, you straighten out the system that lets people actually sign up. If it takes 6 more months to build a robust sign-up system, that is a PR issue that can be managed, as long as the tire-kicking” system would enable consumers to satisfy themselves about what Obamacare means to them personally.
3. There has always been a high failure rate in big software projects. This is true in the private sector as well as in government. In this case, I think a lot depends on how easy it is to take out one part of the system and fix it without affecting other parts of the system. If the internal interdependencies are too large and complex, trying to patch the system can turn out to be a bigger challenge than starting over by creating a simpler architecture.
The biggest issue with integration of many large databases is the data. Let’s say RI calls something one thing, MA calls it another, while NH collects different but similar data and uses it for the same purpose, etc. With all DBs across all the states, corporations, fed government and who knows what else they are trying to pull in, there are a lot of differences. Now, multiply all the differences for each interface times the number of interfaces, and that is your complexity. But that is not worst of it, bugs in first interfaces undoubtedly cascade down. After you put the system in and find a bug, fixing it requires addressing all the special cases and inevitably you break one or more of the special cases especially down stream. Panic fixing, like I imagine is going on now, never yields good results.
I think the complexity of these apps are being overstated, but they have enough going on to require a certain higher level of architecture, programming and managerial talent. Average IT talent just won’t cut it for this class of application.
Unfortunately, I can’t recall ever seeing a web app done by a government entity that was impressive technically.
Can anyone cite an example of a government website that impressed them?
Back in the 1990s, the Department of Energy had a web site that was pretty good for that era
Look up the various agencies that have tried to implement a LIMS (laboratory information management system). FBI has tried several times (for their crime lab, integrating with various state agancies), never successful. They threw a lot of money a public health LIMS after 911, with varying success, but with a vary scaled back integration with a common protocol already in existence (HL7). Running a web site is easy, integrating systems is HARD.
As an expert on development of sophisticated web sites, I can answer with authority that your understanding of #1 is correct.
Of course I haven’t seen the code, but from what we can gather about it as a “black box”, there is every appearance that the problems are fairly fundamental to the underlying design of the system. Tweaks to the implementation, as could be accomplished over the course of a few days, won’t be have the kind of fundamental difference in the way the system operates that seem to be required.
If the system is to become usable, it will be due to one of two things:
A) Tearing down a large part of the actual implementation and rebuilding it with a different approach that considers its scalability as a fundamental operating parameter.
B) (Attempted) usage of the system declines sufficiently that the system is able to cope with it. You can’t just say that if the usage falls by, say, 75%, that the performance will improve four-fold. These kinds of scaling problems tend to be non-linear. As the usage increases, you tend to hit an inflection point where the behavior of the system just goes to hell. If the number of people accessing the site falls below that threshold, it may be OK. (Of course, then they’ll trumped “it works”, and everyone will jump back in, thus bringing it back down to its knees).
Of course, given early reports of premiums and deductibles relative to what people already have now and are losing, to allow “tire-kicking” before registration would probably scare away many tire-kickers once they’ve seen the numbers. Hence the government’s desire for registration first.
Oh, but this is silly! We’re not exactly two weeks into the Computer Age, after all. We’re not even just two weeks into the Era Of The Internet. We’ve got some large database systems on line: Amazon.com for example, or nearer at hand, the Social Security website. We know how to do this crap!
Yes, I understand that preserving user confidentiality is Terribly Important — although in a world run by the NSA, I have to snicker a bit — and that at some point user inputs need to verified (is that the right SS number? is that the right salary figure? Is the number of children correct, since you’ve told the IRS something quite different for three years now?) But the user interface part of this — putting up clearly worded instructions in several languages, showing options, collecting and recording a few fields of data, displaying the results and asking for confirmation — should have been duck soup, and that’s the part. that seems to gone most badly. People ought to be getting fired right and left for incompetence here — and not just low level programmers.
Interesting piece in the WaPo (http://www.washingtonpost.com/blogs/wonkblog/wp/2013/10/09/healthcare-gov-was-originally-built-in-a-garage/)
which indicates I got this half right. Tasks were indeed split in a modular fashion, with the user interface being one part, and the backend verifications and contract making in another, with separate firms working on the modules. However, while I’d assumed from my reading that people were having problems with the front end UI stuff, the Post is putting the blame on the back end stuff that I’d regarded as straightforward.
Hmmm.
(I recommend the Post story for the curious; you might want to read it with a mental grain of salt, since it quotes one of the two programming firms liberally, and has no quotes from the other.)
Who, or what entity, produced the Product Performance Specification for this albatross? Or is there even a Product Performance Specification?
(Note: NOT a Design Spec, but a Product Performance Spec. Design Specs follow from the PPS.)
Or was this mess kluged together using the Nancy Pelosi theory of systems engineering – “We have to build it before we can find out what’s in it”?