In his new book, Tim O’Reilly reminisces fondly about the origins of “open source” software, which he dates to 1998. Well he might, for his publishing company made much of its fortune selling books about various open source languages.
In contrast, in April of 1999, I called open source The User Disenfranchisement Movement.
…The ultimate appeal of “open source” is not the ability to overthrow Microsoft. It is not to implement some socialist utopian ideal in which idealism replaces greed. The allure of the “open source” movement is the way that it dismisses that most irksome character, the ordinary user.
In that essay, I wrongly predicted that web servers would be taken over by proprietary software. But that is because I wrongly predicted that ordinary civilians would run web servers. Otherwise, that essay holds up. In the consumer market, you see Windows and MacOS, not Linux.
The way that open source developers are less accountable to end users is reminiscent of the way that non-profit organizations are less accountable to their clients. Take away the profit motive, and you reduce accountability to the people you are supposed to be serving.
Still, the business environment is conducive to firms trying to expose more of their software outside the firm. When a major business need is to exchange data with outside entities, you do not want your proprietary software to be a barrier to doing that.
A local college computer teacher, whose name I have forgotten (I briefly hired him as a consultant but fired him quickly because he was disruptive) used to make the outstanding point that the essential core of computer programming is parsing. There is a sense in which pretty much every chunk of computer code does the job of extracting the characters in a string and doing something with them.
Computer programs don’t work by magic. They work by parsing. In principle, you can reverse engineer any program without having to see the code. Just watch what it takes in and what it spits out. In fact, the code itself is often inscrutable to any person who did not recently work on it.
Ironically, some of the most inscrutable code of all is written in Perl, the open-source language that was a big hit in the late 1990s and was an O’Reilly fave. If you want to reverse-engineer someone else’s Perl script (or your own if it’s been more than a couple of years since you worked on it), examining the code just wastes your time.
There are just two types of protection for proprietary software. One is complexity. Some software, like Microsoft Windows or Apple IOS, is so complex that it would be crazy to try to reverse engineer it. The other form of protection is legal. You can file for a patent for your software and then sue anybody who comes up with something similar. Amazon famously tried to do that with its “click here to order” button.
In today’s world, with data exchange such a crucial business function, you do not want to hide all of your information systems.
You want to expose a big chunk of them to partners and consumers. The trick is to expose your software to other folks in ways that encourage them to enhance its value rather than steal its value. Over the past twenty years, the increase in the extent to which corporations use software that is not fully hidden is a reflection of the increase in data sharing in the business environment, not to some magical properties of open source software.
Android is the most popular user operating system in the world with around 2Bn users, and it is Linux based and the core is open source.
But the bits that now really matter – various Google services – are proprietary. There are no successful smartphones that run non-Google flavors of Android. Amazon tried that with the Fire Phone, and it was a massive failure.
Years ago in Cambridge, Massachusetts, there was a wonderful institution called the Bicycle Repair Collective. You could use their workspace and tools, they would help you fix and maintain your bike, and numerous other things (including just fixing it for you). All for very reasonable amounts of money. The people who had run it were moving on and a new group, which included one of my housemates, was taking over. They had an organizational meeting at our apartment and since I had used the place, I was kind of part of it (it was all pretty ’70s informal).
The new people were all committed lefties and when the topic of hours came up, they were unanimous: workers deserved weekends without work, so the place would be closed Saturday and Sunday (which would be a change). I pointed out that the customers were mostly workers, too, and that often it was hard to go in the evening; Saturday and Sunday were the best days for most of them to use the Collective.
I wouldn’t say the members were shocked but it was something they simply had not considered and it took them a while to accept that they really shouldn’t close all weekend. Finally, they agreed to open one weekend day.
O’Reilly is stuck on the idea of post-growth.
The appeal of open source for developers is the ability to find and share solutions to common technical problems. Something like Ruby on Rails allows a single developer to focus on user problems instead of dealing with the technical necessities of connecting a web page to a database without paying a license fee to anyone.
No, he’s really very close to a fundamental truth – not having to suffer the end users saves a great deal of time, trouble and heartache. Even not having to make a GUI out of something probably knocks the cost way down.
I think open source long ago gave up trying to be mainstream. Linux dominates the internet, someone mentioned Android, but most network hardware (e.g. routers) are Linux – might be all. Throw in servers and the internet would not exist without Linux. I run Linux on all my computers (Centos and Ubuntu) and the parts of Ubuntu that irritate me are the parts that they try to make user friendly. Open source is great for a very specific subset of users, because developers understand the needs of other developers.
Don’t forget all the embedded stuff that has linux thrown in to avoid costly programmer expense when they can get a $1 hour programmer from overseas. Unfortunately, this “internet of things” also includes all the security vulnerabilities built in without a means to update the software or even cursory lockdown.
Linux is no way to avoid “costly programmer expense”. I won’t speak specifically to settop boxes and media players ( because they’re disaster areas[1] and nothing seems to matter there ) but everything else? You need people who know what they’re doing.
[1] they are, in essence, an exposure of video driver interfaces which cannot be shown to work much less be proven to work.
“Programmer expense” only matters for projects that probably should not be undertaken at all. As sez Jaron Lanier, “software is people who aren’t there” and it should be screamingly obvious by now that that’s very nearly always cheaper when done properly. I know for a fact that I’m roughly an order of magnitude less expensive for the last 10 years of projects I’ve done on rework alone. You don’t learn that in a boot camp.
Now, modern accounting practice is to construct artificial burdened run rates to lard up the Administrative State portions of firms and then try to pin that on the software guys, but I can tell you that I am just as effective without all that as I am with it.
The fact is – it takes *ten to twenty years* to reach master status in software. Don’t let people tell you otherwise – sure, you’ll be productive in five, but you’ll still leave defects that will cost a lot of money in.
We can write software, we can account for the cost of that activity but we can’t do both a the same time.
As a programmer, I think you’re wrong about data exchange, but half-right and half-wrong about open source.
First, modern data exchange is driven by open formats and open APIs, especially REST (basically following HTTP principles for websites for data exchange ). This actually abstracts away the underlying software. Open APIs can be implemented by propriety software or open source software, and it does not matter. In my opinion, modern data exchange is orthogonal to open source, and is more driven by the fact that we saw how well HTTP works to visit many different websites, and adapted those principles for general data exchange.
For open source itself, I believe there are two types of software: software written for other programmers; and software written for non-programmers. Open source has made a huge impact in the first category, but very little impact in the second category.
Perhaps this comes down to the notion of customers. Other programmers are the customers of open source software, and open source developers care about their opinions. Or possibly that the developers are themselves the customers, because they use that software elsewhere. So open source software written for other programmers is extraordinarily good. Meanwhile open source software written for non-programmers is pretty bad, because the developers aren’t accountable to the customers, as you note.
There was too much bouncing of the software rubble in those days. Too many ways to accomplish the same task. So the market concept was platform, a universal method to organize coordinated protocols. Google search, Windows SQL, the browsers; all became platforms and adopted systematic methods to enhance access and build new applications.
I can only guess at the damage done by SQL alone.
“The great thing about Linux is that you can do anything with it. The problem with Linux is that you must be an expert to do anything.”
Expertise in Linux is relatively trivial – until it isn’t.
You really think the essay “holds up?” What about this part?
This is one of the few (only?) specific predictions in the essay and, to put it kindly, it did not hold up.
Moreover, another type of software, that you did not mention but whose properties are very much like the web servers you do mention, is statistical software. Back in 1999, this area was totally dominated by proprietary software: SAS, MatLab, SPSS, et cetera. It is now, almost, as dominated by open source, mainly R and Python.
In that essay, your concluding summary seems to be:
Hasn’t this proven 100% false? Isn’t open source software much more popular/successful — using any definition of popularity or success you like — than it was in 1999?
Of course, I am somewhat biased!
I remember running one of the first spreadsheet programs on a VAX, well before PC’s. It came with a “big boy warranty.” As in if anything goes wrong, you’re a big boy now and get to fix it yourself.
“[T]he essential core of computer programming is parsing” in much the same way that the essential core of economics is balancing equations. Which is to say, it is only true for someone trying to seem witty while being grossly inaccurate. Parsing is at most a part of the input problem. Google’s search and advertising algorithms, MPEG video (de-)compression standards, industry-standard 3D rendering processes, faddish “deep learning” networks, and most other marquee software only starts being interesting after the input is parsed. That is not to say that a software developer can afford to be sloppy with parsing, but it is fundamentally no more important to the endeavor than finding the right equations in an economic model is to a broader economic analysis.
Successful open source software seldom tries to solve the needs of the general public; as a consequence, open source desktop environments are generally awful and do not appeal to a broad audience. Most often, successful open source software tries to scratch an itch for the people who write the software. Sometimes the developer is an individual hobbyist; now, it is often a person or team whose business builds on the solution provided by the software.
In that respect, I think open source software is a stable and productive slice of the software ecosystem because increasing levels of specialization — both at the individual programmer level and at team and even company-wide levels — mean that cooperating to solve collateral problems is rather directly driven by considerations of how to efficiently use resources, and that more subject matter will become “collateral” in the sense of being a prerequisite or tangent to a main line of business.
Despite the dreamy hopes of leftist techies, Open Source is not a socialist paradise for programmers.
The free market is a failure machine. There are a lot of dead open source projects out on GitHub. Without a vibrant user and developer community, those projects die just as any voluntary market venture does.
It is very much a free market system. It just doesn’t involve currency the the way that we’re used to.
Inside a company, the tense relationship between developers and users is usually a less market driven relationship.
“Dealing with users is difficult. They come up with requirements that wreck your underlying data model. They come up with ways of using your software that expose weakness in its architecture.”
This model is sloppy because:
1. Users don’t have to directly pay for developers’ time.
2. Developers don’t make more money having to re-architect software on users’ whims.
There are costs, of course, but they don’t easily show up in budgets and in the accounting department’s reports.
I worked as a contract developer for a few years during the .com boom alongside salaried developers. Because I was paid hourly, my employer was much more thoughtful about requirements that I was given.
On the one hand, I was under more scrutiny by management regarding my value to the company. On the other, my time was respected more. While I worked 40 hours most weeks, the salaried workers had to put in many more frustrating hours.
Of course, when the dot com bubble began to burst, I was the first to pack up my desk!
By the way, I agree with the essay that the appeal of open source to some programmers is the idea of avoiding users.
It does seem, however, that the most successful open source projects do depend on and are most responsive to strong user feedback.