Photo: Flickr user lifeontheedge

Thursday, August 04, 2005

The Future of Open Source

5 years ago Linux was cool (coolness propogated down from the highest levels of geekery to the merely internet-addicted, who absorb it eventually through long exposure, like they do web design); now some developers are deserting -- proprietary OS X gives them a command line, and that's all they really want.

At least in the first world, the thrill is gone.* Most people are more concerned with operating systems as tools than as ends in themselves; tinkering has lost its allure; programming should focus on rewarding, important problems, not getting your sound card to play two sounds at the same time.

(Tangent. In the 3rd world, unnoticed on Slashdot, Linux could be really, really fucking important. Commodity hardware, newly inexpensive; free software. People using computers to look up crop prices. Small towns using local builds customized in the local language. (No sound, no video-in, but who cares?) A homegrown software industry. Unglamorous stuff, but far more important than how cool your G4 Tower looks.)

Sever-side linux is still a powerful force, but what happened to the desktop utopia that was supposed to unseat Windows? And will the same developed-world disenchantment hit Wikipedia as it grows?

While in principle anyone can contribute to an open source project, Linux's barriers to entry are higher than Wikipedia's. Even correcting minor Linux bugs is well beyond (for example) my expertise, but my grandmother could edit Wikipedia. All you need is an internet connection and literacy (mid-level literacy, at that; other people will fix your grammar and spelling).

Wikipedia can draw on half a billion potential contributors; only about 100,000 people can code Linux.

It's hard to overstate this difference.

This is about where people start throwing around terms like "emergent behavior" and "paradigm shift". Over the next few years, Wikipedia (and some of its Wikimedia sister sites) will become comparable to Linux in economic and social significance. (Maybe Linux will catch up again a few decades later, if schools start teaching as many kids to program as they teach to write.)

(Note: I'm not saying wikis in general are going to be terribly revolutionary -- making a good, scalable wiki is about as hard as setting up a democracy in the middle east. And momentum is vital: like Ebay and Craigslist, wikis are only visited if they're contributed to and only contributed to if they're visited. Wikitravel might yet find its legs, but right now it's the Betamax to Wikipedia's VHS -- Wikipedia has much more info on most locations.)

Of course, the picture is very different if you look at the number of actual contributors:

56 thousand people have edited wikipedia and about a hundred thousand (though this number is difficult to estimate) have worked on the Linux kernel and its best-known desktop environments. (Other Linux applications aren't counted, and neither are wikipedia's sister projects.)

But wikipiedia is very young: less than a third the age of Linux. Compare the most influential contributors (so that linux data is available) over time and the picture changes again:

Those are only the most die-hard fans -- Wikipedians with more than 100 edits a month and Linux developers in the Credits file. But the huge upward sweep is exactly what graph #1 would lead one would expect.


Software mavens are tinkerers by nature. How many people who use Wikipedia will actually edit it?

There's a large gap between the skill necessary to run linux and (for example) to compile it, and a similar gap between compilation and real, gritty coding. Wikipedia's learning curve is shallower: you can reorganize and reformat pages, yes; you can contribute esoteric trivia to articles on history and philosophy and find diplomatic dispute resolutions. But you can also correct a typo in an article about your city.

Some fun articles to write are not particularly useful, and vice versa. Linux has been saddled with this problem: volenteers ignore boring but necessary work.

Scale solves this one -- yes, there will be mountains of fun-to-write content on wikiepdia (it's possibly the world's best Harry Potter resource). But there are so many contributors that some of them will want to write detailed histories of the Democratic Republic of Congo and help compile comprehensive lists of the etymologies of the names of Ohio counties.

Won't wikipedia deteriorate as it gets bigger?

It's gotten a lot bigger in the past few years, and much better as well -- if wikipedia can scale from 50 to 50 thousand contributors (and scale superbly), it can probably scale from 50 thousand to 50 million. Wikipedia is not anarchy; it's a complex system with many subtle but well-honed rules and balances.

Wikipedia contributors don't work together -- pages are hacked out through interminable debate; huge amounts of effort are wasted on minutia.

I think this is the exception rather than the rule. But more importantly: huge amounts of effort may be expended, but only in sum; the effort is distributed with incredible efficiency -- anyone can put in five minutes, or less. 50,000 people, times five minutes a day of labor, times 30 days, is 125 thousand man-hours: 14 years of continuous labor, or about 60 years of 40-hour weeks. Even if 90 percent of that is reverted (an absurdly pessimistic estimate), it's still more than 6 years' work completed in a month. Wikipedia is one of the few places where such small individual contributions can be easily combined. (This brings up another point: people work much more efficiently in shorter increments.)

The important thing is that people aren't made to feel their work has been wasted -- nothing kills the contributory spirit like having your painstaking paragraphs reverted. This is a problem that needs a social solution, and hopefully one will evolve/spring up.

Wikipedia and Linux are completely different! They certainly aren't competitors; wikipedia runs on linux.

They're not competitors, but they're both open-source projects. Open source is an important organizational framework that until wikipedia had been used mostly for programming, not writing human-language text. The most general point is taht the visibility and importance of open source writing will eclipse the visibility and importance of open source programming.

Linux isn't just the kernal and desktop environments. What about all the software?

Of course, there are hundreds of other programs for linux, but there are hundreds of potential structures, programmatic, community-based, and otherwise, that can run off the mediawiki base; most just haven't appeared yet because wikipedia is only three years old.

Linux bugfixes aren't that hard to code.

You've obviously never tried to teach basic HTML to nursing students. It's easy to overestimate the technical savvy of most people, especially when you hang out with programmers.

Your 3rd graph sucks.

You're right, it does. But I think the reasons it sucks counterbalance each other.

  • More people have contributed the linux kernal than are listed in the Credits file

  • Linux actually began in 1991; 1994 is just the first year the Credits file was published -- effectively, the graph gives linux a 3-year head start.

  • The credits file is cumulative: if you're in it once, you'll be there forever. The number of very active wikipedians is monthly: edit like mad in August, and you're counted for August -- drop below the threshold when you go back to school, and you're off this list.

*Desktop Linux isn't fading away! (section added June 2006)

I agree, and Ubuntu seems to be gaining steam. But I remember when people were seriously talking about windows getting unseated. Hell, Firefox hasn't even unseated IE for 80% of the market -- but Wikipedia's unseated Britannica and World Book as the most widely read encyclopedia.

Because I'm addicted to pageviews, you should share this article:, Digg, Furl, ma.gnolia, reddit, Spurl, Yahoo.


Anonymous said...

Wikipedia, provided no major disaster occurs, is destined to be a permanent and important human institution long after all the copyrights on all the operating systems run out. Its place in intellectual history will be comparable to Academy. It is not really an encyclopedia. It is sui generis, a new thing, an institutional form comparable to the university. Look ahead to what it, and related projects will be in 20 years, 100 years.

Anonymous said...

Wikipedia definitely has the potential to be much, much more important than most people seem to think. The strengths of Wikipedia outweigh the drawbacks, and with some years of development I think most people will come to the realization that regular encyclopedias are outdated. Put the words "DON'T PANIC" on Wikipedias frontpage in big, friendly letters and the deal is settled.

Anonymous said...

Wikipedia is already moving beyond usefulness for the layperson in many precise topic areas. It is unlikely this trend will reverse itself, although there are efforts currently underway to counteract it. There are also efforts to limit the intrusions of newbies into the database, to channel their efforts into established article templates, to create "security" systems to manage their creative (if often misguided) attempts to help out.

At one point Linux, too, was more readily accessible to laypeople.

The universal trend is toward complexity.

Ben Yates said...

That's a good point (and one I hadn't thought of). But I think accessibility can be preserved, if things are taken carefully enough. "The universal trend is toward complexity" is, I think, over-pat.

For example, the brief encounters with wikipedia I had a couple years ago were not just unimpressive but uncompelling -- the articles were so incomplete that I didn't have any urge to contribute to them. Now that the articles are more full-formed, contributing is kind of addictive.

Ben Yates said...

Alright, on further reflection: yes, I wouldn't be surprised if wikipedia became much less fluid and accessible. But it still has the advantage of scale (graph 1), so its maturation should be very different from Linux's.

Anonymous said...

I really think Wikitravel is going to make it.

Its obviously laking right now but I find it an incredibly useful tool for finding information about places to go and even things to do in my own hometown.

Anonymous said...

You raise some interesting points, but your numbers are fishy. You say "only about 100,000 people can code Linux" and then you say that "about a hundred thousand (though this number is difficult to estimate) have worked on the Linux kernel and its best-known desktop environments."

Certainly one couldn't say that every programmer capable of working on Linux has. How do you justify the same number for both statistics?

Unknown said...

Your comparison is off, I'm afraid. Adding wikipedia entries cannot be compared to writing software programs. Anyone can contribute to "Linux." But not anyone can contribute "code" to "Linux." Unless you are someone trained in the art of coding and decoding patterns in software programs you cannot contribute code to free software in general.

But placing the skill required to code and decode patterns in software programs on the same level as adding new entries to wikipedia is unfair. It takes years to be proficient in programming and much longer to be an adept software developer. It takes less than a day to two to figure how to add entries to wikipedia, if you are computer literate.

In the free software community, software developers are overrated, in my opinion. I understand why. They are scarce. But in the free software ecosystem, other contributors are as important if not more important than the software developers. Testers, documentation writers, translators, artists, project managers, fanatics, etc all comprise the free software community. Anybody can be any of those in some capacity, and the barrier to entry for most of those is not as harsh as you make it seem.

The problem is there is a misconception that the only way to contribute to free software projects is by contributing code. Your blog entry confirms this misconception.

Anonymous said...

Recently, i have found, at least for the searches that i make, that wikipedia is a more relevant source of information than the collective results from google (at least those results that i cared to browse through).
I think it is competitive, and can only imagine the results that emerge as more people get on board to contribute. It is quite interesting to note that as our machines get more intelligent, the premier source is still man-made.

Anonymous said...

My article Disorganized Incremental Software Development is on a similar theme, although it implies that in order to make software as easy to add to as Wikipedia, you actually have to raise the bar: use formal methods, and program against formal specifications.

Ben Yates said...

Phil's link didn't work; I think he meant to link to this.

Ben Yates said...

The best commentary on this post that I've read.

Anonymous said...

programing is not the same that writing. In Wikipedia everione can edit... but how many people know about assembler or programming? said...