Clay Johnson and Harper Reed write,

HealthCare.gov needs to be fixed. We believe that in a few days it will be.

A few days?

They suggest that the government should modernize its procurement practices and development methods. On that issue, I do not disagree. But such modernization is not a magic bullet.

Johnson and Reed have experience in using the Internet to help political candidates mobilize supporters. To me, that is not the same thing as building a complex, mission-critical system to support a business.

What I am pushing is the idea that success in building such a system is not a matter of finding a technological magic bullet. It is a matter of starting with the right kind of organization on the business side. If the government is going to operate the world’s biggest health insurance brokerage, then the government needs to set up a business organization to manage it, with clear lines of authority and communication. Yes, it doesn’t hurt to use the most effective development tools and methods. But show me an ineffective org chart in the business, and I’ll show you a systems project failure no matter what tools and methods they use.

[UPDATE].

Jeff Sients, the new Suit, is not talking in terms of days, but he still seems optimistic.

Zients said healthcare.gov will be functioning “smoothly” for the vast majority of users by the end of November

The Suits are now talking about a “punch list,” as though this is a house that you just bought and the builder has to touch up the paint in a few places. For all I know, that accurately describes the situation. The worry is that instead of a basically solid building they have another Silver Spring Transit Center.

Ezra Klein interviews Fred Trotter, a health IT specialist who steers away from government contracts. Some excerpts:

I realized I could figure out how to develop these very complex, very new software programs or I could figure out how to contract with the government…One of the jokes we have in health IT that doctors have no idea what they want and we’ve been giving it to them for years. And I think that’s a fair assessment of technology procurement. The government has no idea what it wants and contractors have been giving it to them for years…There’s a problem in government IT departments in general where you get somebody who got a job in their 20s and didn’t really have any reason to improve their skill set or change their approach over 20 years but now they’re in charge of a department. People think if you are a geek and a technologist and that’s the way it is. But if I knew what I knew four years ago today and that’s all I knew today I’d be out of a job.

The New York Times reports,

One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly.

1. There is zero chance that rewriting five million lines of code is the answer. Either the solution is a lot simpler or there is no solution other than to start over.

My instinct from the outset was that starting over was the right answer. I am not alone.

2. The other day, President Obama said, “No one is madder about the Web site than I am, which means it’s going to get fixed.”

Or, as Michael Palin and John Cleese would put it, Wake up, Polly!

3. In response to the WaPo story, I wrote a letter to the editor, which they published (mine is the third letter on this page). This is not a technical screw-up, and it will not be fixed by technical people. It is an organizational screw-up. And until that is recognized, it probably will get worse. I write,

In my experience, communication failures between technical staff and management reflect an atmosphere of fear and lack of mutual respect.

I call this the suits-geeks divide. I saw it during the financial crisis, when it was evident that many mortgage credit-risk geeks warned of problems at their firms but management went out of the way not to listen. Merrill Lynch and Freddie Mac were particularly well-documented cases.

4. Every year, I have my high school students pair off and present proposals to start a new business. Two questions I always ask are “What are the critical management functions?” and “What would somebody experienced in this business know that you do not know?”

Suppose that President Obama and Secretary Sebelius were in the class, and they proposed starting the world’s largest health insurance brokerage. I would expect them to be able to identify as key management functions: marketing, customer education, insurance company partnership management, pricing and underwriting standards, and operations.

Somebody who had experience with creating a health insurance brokerage business would know that the systems problems are more complicated than just putting up a web site. In the background, the system needs to communicate with the systems at several government agencies and at the insurance companies. That changes it from a simple technical project to a complex, time-consuming, project involving business and technical staff.

You build a complex, mission-critical system through a process of continual negotiations among business units and technical people. You do not treat it as a procurement process. You cannot just write up a spec, put it up for bid, and parcel it out to dozens of contractors.

The development of the computer system probably would fall under operations, but you would want a project executive with a lot of authority to negotiate with all of the business units and to make project decisions. When conflicts arise, the project executive should be able to go straight to the CEO and get them resolved.

The project executive’s main focus is keeping the project’s complexity from getting out of control. The project executive must have the authority to trim features in order to meet deadlines.

You go through a lot of analysis and many painful meetings before anyone writes a line of code. The technical staff have to be able to challenge the business units, because sometimes the business unit asks for something to be done in a really complicated way, when a much simpler solution is available to solve the business problem.

One of the worst things that can happen on a systems project is to find yourself revisiting the business-technical negotiations process after writing a lot of code. If that is what is happening now, this project is in an unbelievable amount of trouble.

5. I suspect that the technical problems are mere symptoms. Probably what is fundamentally messed up in this health insurance brokerage business is the org chart.

6. In business, you need clear lines of authority and accountability. The bureaucratic tendency is to seek the opposite–to blur authority and avoid blame. This is a big challenge in the private sector. However, I think it tends to be even more difficult to overcome in government.

7. For Christmas, someone should give President Obama and Secretary Sebelius a copy of The Mythical Man-month.

[update: good PBS interview with several mythical-man-month allusions]

[update: Clay Shirky’s tweet echoes this theme.

From the book Lost Moon, retitled Apollo 13 after the movie was made:

Apollo was downright dangerous. Earlier in the development and testing of the craft, the nozzle of the ship’s giant engine…shattered like a teacup when engineers tried to fire it. During a splashdown test, the heat shield of the craft had split open, causing the command module to sink like a $35 million anvil to the bottom of a factory test pool. The environmental control system had already logged 200 individual failures the spacecraft as a whole had accumulated roughly 20,000.

In January 1967, one of the first Apollo spacecraft caught fire during an on-the-ground test, killing astronauts Gus Grissom, Ed White, and Roger Chaffee. At that point, NASA decided that quality was more important than schedule and they overhauled the Apollo project (although they still managed a moon landing 2-1/2 years later).

In some ways, the rollout of the Obamacare web site is reminiscent of this. The hurried schedule appears to have hurt quality. The rational thing to do now would be to let the schedule slip and resolve the quality issues.

Speaking of which, here is one recent report.

Recent changes have made the exchanges easier to use, but they still require clearing the computer’s cache several times, stopping a pop-up blocker, talking to people via Web chat who suggest waiting until the server is not busy, opening links in new windows and clicking on every available possibility on a page in the hopes of not receiving an error message. With those changes, it took one hour to navigate the HealthCare.gov enrollment process Wednesday.

Those steps shouldn’t be necessary, experts said.

Neither was the last sentence.

Megan McArdle writes,

I predicted in December 2012 that the exchanges would not be up and running on time with minimal knowledge of how the contracts and budgeting were being run, because the administration was being pretty closed-mouthed about those things. Was I prophetic? Hardly. I just didn’t see how the administration could make things work in the allotted time frame. The development cycle was just too aggressive, even with what my boss used to call a “Shake and Bake” system (take something out of the box, add a few of your own ingredients and roll it out). I thought about the software-implementation projects I’d worked on (not in development but on the server side) back in the days when I was an IT consultant. This seemed a lot faster than anything that any company I’d ever worked with would commit to, even if it had already designed some of the underlying architecture.

Michael Malone writes,

Anyone who has ever worked on or, worse, bought a big software application – and this is one of the biggest in history — could have told HHS that the final result would be buggy, late, unsatisfying to users, unable to live up to its billing, and most of all, resistant to upgrades, much less wholesale changes. In the real world, you can’t just order “Make it so!”

I, too, have some experience with software development. This sounds to me like a case of suits saying one thing and geeks saying another. The suits are talking about “glitches” and “too much demand.” The geeks are making it sound like the system might be FUBAR.

1. I have been reading that the process of interacting with databases in order to deal with user identity and security issues is cumbersome, and that is causing the problems, even at small scale. If so, then this cannot be solved by adding servers or by patching a few lines of code. It may be closer to “we need to rip the guts out of the segment of the system that manages data transfer and re-architect it.”

2. My instinct at this point would be to try to separate the functions involved with sign-up from the functions involved with “tire-kicking.” I would think that it would be easier to set up a “tire-kicking” site where people can find out what the plans are and approximately what they will cost given their personal situations. However, because you could not actually sign up, the system would not store personal information or access so much data. Meanwhile, you straighten out the system that lets people actually sign up. If it takes 6 more months to build a robust sign-up system, that is a PR issue that can be managed, as long as the tire-kicking” system would enable consumers to satisfy themselves about what Obamacare means to them personally.

3. There has always been a high failure rate in big software projects. This is true in the private sector as well as in government. In this case, I think a lot depends on how easy it is to take out one part of the system and fix it without affecting other parts of the system. If the internal interdependencies are too large and complex, trying to patch the system can turn out to be a bigger challenge than starting over by creating a simpler architecture.

A reader points me to this story.

What we want to do is create an appropriate scale currency for our region that will then serve the needs of our region. We used to have a system in this country of local currencies everywhere, and then there was a national currency, too. So that’s what we’d like to see again: regional currencies that work for their region and then a national currency — why not? You need that too so that you can trade across the country, or even an international currency.

I see this as analogous to frequent-flyer miles. It is a pseudo-currency that is worth something in some contexts but otherwise is not very liquid and is close to useless. As with frequent-flyer miles, the idea is to encourage customer loyalty, in this case to local merchants. I would be surprised if it turns out to be important.

Megan McArdle writes,

Costco really is a store where affluent, high-socioeconomic status households occasionally buy huge quantities of goods on the cheap: That’s Costco’s business strategy (which is why its stores are pretty much found in affluent near-in suburbs). Wal-Mart, however, is mostly a store where low-income people do their everyday shopping.

I went to Costco for the first time a couple of months ago. My first reaction was that I did not understand the business model. I thought that in the grocery business, you wanted to have lean inventories, and they seem to have the opposite. McArdle explains,

Costco has a tiny number of SKUs in a huge store — and consequently, has half as many employees per square foot of store. Their model is less labor intensive, which is to say, it has higher labor productivity. Which makes it unsurprising that they pay their employees more.

My local grocery store, Giant, is filled with employees, who are constantly restocking shelves. Giant keeps on hand relatively low inventories of a much larger number of products.

Eventually, I could imagine an equilibrium in which a store like Giant pares back on the number of items it sells in the store, keeping only the most popular items available. You would have to order less-popular items on line. That way, they could cut back on those restocking costs.

Of course, for all I know, Giant’s business model is to charge a big markup on stuff, and they figure once they get you into the store they make a profit. And if stocking a great variety of items gets you into the store, then that is the right strategy.