> The only thing that distinguishes the Cathedral and the Bazaar is whether or not there is one single individual in whose head the only valid vision of the completed project exists.

Core OS and related groups has long consisted of VERY different teams each working in their own fiefdoms with their own methodologies. There was a somewhat coherent vision overall, but everybody achieved it in their own way, with very mixed results -- OS X has some tremendously terrible code. Security framework, Installer team, mDNS, anything related to Server ...

To be honest, I'm not even sure what your point actually is.

The point is that, if one of the workers building the Notre Dame had gotten hot after a day of work and said "You know what we should include here? A swimming pool!" well...he probably would've been committed.

Coherent vision and a priori design are not the same thing. Apple has coherent vision. Bazaars can have coherent vision (should have coherent vision if they hope to be successful). But that's not the same thing as a Cathedral's a priori design...

A priori design can adapt to new ideas. I'd argue that's what Apple did/does, in many cases.

Likewise, I'd argue that the truly ad-hoc bazaar development is responsible for some of the worst ideas and bad code that can be found at Apple.

The historical lack of good centralized vision on aspects of the Core OS -- such as Objective-C -- has led to staggering missteps and ridiculous inefficiencies on behalf of both the framework and compiler teams. This has been to nobody's benefit and the sum result is clearly inferior to better-designed language work done elsewhere (eg, MS).

Likewise, the ability for applications teams to drive forward ill-conceived OS and framework hacks has led to some terrible long-lasting implementation failures, which is something a coherent top-down vision could have prevented.

However, the fact is that products can succeed despite of their poor implementation. Costs may be higher, bug counts may be higher, and user satisfaction may be lower, but that hasn't always stopped Apple from building successful products. Where I take umbrage is in the notion that there's a dichotomy -- either you do things poorly and let intellectual lazy engineers take the lead, or your product does not succeed. That's not accurate.

I don't think Apple is a good case study for your point.

Compare FreeBSD kernel design versus Linux.

kqueue vs. dnotify/inotify/???

Mach-descended VM vs. a string of linux-vms

BSD scheduler, ULE scheduler vs. how many different schedulers?

Linux churns through ill-conceived solutions to problems until they find one acceptable enough. FreeBSD grinds on one until it definitively works.

FreeBSD almost invariably winds up with the better solution, in less time. See kqueue, for example -- the foundation upon which Apple's GCD is built.

Planning ahead can lead to great things. The Notre Dame, and the dozens of other cathedrals throughout Europe are positively stunning...

I love and I hate kqueue. I mean, I love kqueue. I love the way it can be used from C, I love the way it's integrated in MacRuby...I spent the weekend studying Clojure's reducers and was dying to have some time to work on ClojureC just so I could implement reducers with kqueue...

I hate kqueue because, in all likelihood, I'll never get to use it in a production system, because it's not in Linux.

Cathedrals can be nice to look at. Bazaars are often more functional.

Except that FreeBSD is functional in production, so what is the actual problem? That market effects and accidents of history resulted in Linux becoming more widely adopted?

What does that argue for, exactly?

http://en.wikipedia.org/wiki/USL_v._BSDi was hugely crippling at a very critical moment.

In a similar vein, PostgreSQL lost out to MySQL in no small part due to:

- PHP supported MySQL out of the box.

- MySQL was slightly easier to get running.

If there's a lesson we ought to learn, it's that market success may be partially or fully disassociated from actual merit relative to other market entrants. We've seen this in commercial software. It would be foolish to think it doesn't apply to open-source.

And what happens if it never works?

libkse was an attempt to implement M:N thread scheduling. This is something that can, in theory, provide significant benefits over 1:1 thread scheduling, but in practice, is extremely complicated to implement and has fallen out of favor across the board: Solaris abandoned the approach in Solaris 9, and FreeBSD abandoned it in (IIRC) FreeBSD 7.

Concurrent to libkse, libthr was developed by David Xu, and was also included in the FreeBSD base system. It implemented 1:1 threading, is far simpler than libkse, and has replaced libkse as the default threading library.

I would argue that in this case, FreeBSD's cathedral model failed; M:N threading ideally would have never been attempted, and it was wasteful to attempt to implement two distinct threading libraries. However, considerable thought and expertise went into the work, and the work (and decisions around it) were not made flippantly or taken lightly.

It was simple a case where despite best intentions and best effort, the wrong choice was made. At the same time, in what some might call "bazaar-like", David Xu maintained libthr as an alternative. It remained there, ready for adoption as the default threading library, until such time as it became apparent that M:N via libkse was a dead-end.

This was a mistake, but no entity is perfect, and the success rate of FreeBSD's considered decision-making remains statistically high. In this case where something never worked, it was replaced with something equally well-considered and far more successful.

Moreover, compared to Linux's gross missteps with their threading implementations (including such gems as setuid() being thread-local as an implementation side-effect), libkse was a minor fender-bender.

There seems to be some underlying assumptions of omniperfection hung of the "cathedral" meme in these parts. That is simply unfounded in both theory and practice.

Second, cathedral vs. bazaar is not really about governance, but about architecture, and they are two separate power-structures, although most organizations, including FreeBSD, mingle them, to their own disadvantage.

We can argue if attempting M:N was a wise or a factually based decision, but that has nothing to do with cathedral vs. bazaar, certainly not in this particular case: What happened was that M:N came first and that was that.

Only once it clearly transpired that it could not sustain its promises in practice (partly because every thread programmer assumed 1:1) did the actual decision making aparatus of FreeBSD kick into gear.

In Linux, (which is much more cathedral than FreeBSD) or OpenBSD (at the time even more so), that decision would probably have been executed practically instantly, but in FreeBSD which is mostly consensus driven, it took some time (and angry words etc.)

But overall the libkse vs. libthr saga has almost nothing to do with cathedral vs. bazaar, because architecture was not the driving force at any time during that saga.

Indeed, throughout this discussion it has become clear that you regard anything with a sense of design and some amount of quality control as a "Cathedral". If you want to discuss the benefits of design and quality control, that's a perfectly fine discussion to have, but it also seems to be completely orthogonal to the original discussion of cathedrals and bazaars, which was wholly focused on the development process.

In fact, I wonder if your arguments would be better focused on the concept of "craftsmanship" (or lack thereof) in programming...keeping in mind, of course, that both cathedral builders and bazaar artisans have historically had a notion of craftsmanship.

I think it is plausible to describe the kernel as much more Cathedral-like, at least in parts where Linus is very strict about how components should be integrated. I can see where parent could have got this impression.

At the same time, there are parts of kernel that are bazaars. Virtualization/Hypervisor support is the area I'm most familiar with; the various hypervisor vendors basically crammed their paravirt calls and passthrough drivers in, without unifying the interface at all. Yes, Linus and others were very aggressive with maintaining the code quality, but less with the architecture. There are other areas of kernel (particular drivers), where we've seen similar situations.

That said, given your response, I think perhaps still don't understand the meaning behind your cathedral/bazaar dichotomy.

While it's been years since I mucked about in Linux kernel space, I always remember it as being a very open and egalitarian community. Although I never made the attempt to contribute to any of the BSDs, I got the impression that they were much less open to newer contributors and divergences from "accepted wisdom" (this could be put more politely as being more discriminating (as in taste) and careful). That being said, the majority of decisions I was witness to on LKML appeared to be well-considered, and make no mistake that patches and design ideas were not just accepted willy-nilly. While Linux has stumbled across many missteps, I still personally prefer the wild possibilities and endless choices that eventually shake out a solution, at the same time keeping with a distribution (Debian stable) that filters out the issues for the end users (myself included).

I also find very interesting the parallels between this discussion and the "worse is better" essay, where UNIX was supposed to be in the "worse" camp, and now it's Linux.

It definitely is, in my experience. Most importantly, despite the obvious presence of "celebrities," even completely unknown people can jump into the middle of a conversation on an important topic, and if they have their shit together, will be accorded pretty much instant respect and be treated as an equal. There's very little sense of "needing to pay one's dues."

But you definitely have to have your shit together. If you don't, you will be quickly eviscerated.

It's really rather nice, especially compared to many dev communities where there's often much more sense of entrenched factions.

I was hoping someone would bring up OS X and how horrible its cathedral-born API became around 10.5, but I was born in '85 and therefore have no right to speak in this thread.

This article concludes that quality only happens if someone takes responsibility for it. Yes. You'll find no argument on that point from me. But cathedral's are not (edit: should say "not only") about one person taking responsibility for quality. They are about a priori design.

Quality control can happen ex post facto but creativity, once strangled, dies.

It seems to me that quality is about having someone on the top who is willing to give direction and vision to a project, instead of having every person who scratches their own itch full influence. Contributors are great. But letting everyone pull in their own direction doesn't lead to something that feels well engineered.

PHK seems to be calling this vision and cohesive design (a-priori or not) the "cathedral model". I agree strongly that for a project to work well and feel "high quality", it needs some source of a unified idiom.

If the suggestion of the article is that the only way to retain quality is to move back to the "single vision" world of a priori design, I'm sorry...that ship has sailed, the cat is out of the bag...whatever your favorite analogy, PHK is very right that the new generation has gotten used to not having instructions handed to them.

Of course, anyone is free to start a project with a single vision, recruit new members, and do their best to grow the project. I suspect, however, that such an effort would loose out to one that figures out how to develop a coherent vision without the need for a priori design.

Nobody belives in "a priori design" and I somehow doubt that anybody did. Brooks points out that the original publication of the "waterfall model" was meant as "how not to..." and people got that wrong.

But cathedrals are not about a priori design, they are about style, elegance, economy of means and coherency of design.

But my point in the piece is that the lost generation doesn't even know what a cathedral is in the first place, having grown up in the bazaar.

ESRs original prototype for the "Bazaar" was Linux. Do you feel that Linux is lacking in coherency of design? You refer to the dot-com bubble, but the problem with the bubble wasn't, I think, that it was the "Bazaar". Indeed, I don't think the dot-com bubble of the late 90s was characterized by much open source development at all!

It was, rather, consumed with "flashiness" and "wow factor". I definitely see the continued obsession with these things as a problem. I would say that, for example, much of the obsession with Node.js today is a consequence of this obsession. But that isn't a Bazaar.

It's a disco.

This is true. The OP has made several somewhat-arbitrary and vague assertions about what may or may not constitute a "Cathedral", on top of lambasting others for not agreeing with his definition ("Read Brooks Book")

Excerpts from the HN discussion :

"Windows and Office are actually not examples of cathedrals, because the architectural focus of Microsoft was not on software but on a near-monopoly market." -phkamp

"...iOS is very much a Cathedral and has a designer and architect who cares and who is in control." -phkamp

Does Apple not care about achieving market dominance? Did no version of Windows or Office have a lead architect who "cared"?

Frankly, the entire concept of a "Cathedral" - rigidly defined as it appears to be here - is a little nebulous and hand-wavy.

You're calling a ot of people out on using Microsoft as a cathedral example, so I assume you want the original bazaar-vs-cathedral reading of the paper. But if you do that, how can autoconf be a bazaar example, when it's GNU software?

With more open development communities there are tradeoffs, but often the result is that you can do stuff you simply wouldn't have the manpower to do in closed communities.

I think most of us would love it if every library and tool we used was pretty and elegant, but in the end, it's often better to have something, however much you'd ideally wish for it to be prettier.

Morever, a community in which different components are often the result of disparate teams with different thinking probably results in more attention paid to robust and simple interfaces between them, simply because that's the only way you can get everything to work. Systems composed of heterogeneous components with robust and simple interfaces between them, are, I think, a good thing. I suspect they tend to be more future proof than systems with everything designed by one person from the top down, because in the end, change is inevitable, so having to deal with heterogeneity from the beginning is an advantage.

It is not just that M4 is a horribly ugly, badly, inconsistent language. You also have about a million leaky caches in the build process to deal with. Autoconf rewrites the build scripts in three stages, each which is cached and also keeps an autom4te.cache directory around. So what happens is that you update something in the M4 build script and then have to spend hours hunting down why the change didn't change anything because some stupid autoconf cache wasn't flushed.

What autoconf has done is to separate free software developers into two categories. Those who can hack the source (the .c and .h files) and those who understand the "autoconf magic." It should be telling that people refer to it as "magic" - it's a fing build system, not rocket science.

The only reason this ugly piece of software has lived on is because people treat it as some kind of magical device and inertia. Configuring and building software is an easy problem. It gets even easier if you finally decide to drop support for 30 year old unices that no one uses. The autoconf team consistently refuses to do that which why the software wont ever become better.

CMake, SCons, Waf and probably half a dozen more tools all do a much better job configuring software than autoconf ever will. People really need to try those alternatives and I'm sure they will realize autoconf needs to die, die, DIE.

CMake, SCons, etc, all have their pros and cons. They are hardly paragons of virtue though. [Yes, even as a mere "consumer" of CMake (someone who occasionally needs to build packages that use it), I've been bitten and frustrated by it.]

As far as I've found, there really isn't any build tool that really gets everything right -- which suggests that the problem is harder than you suggest. [Although judging from the number of people who try to write their own build tool (most of which languish and eventually die, but cause some pain along the way), it's clear enough that many people think it's a simple problem...]

> autoconf needs to die, die, DIE

Decaf?

It doesn't matter for a "works fine" person how many faults their current tool has or how much smoother "your" tool solves all those problems, they won't listen. Logic doesn't work maybe because it is very hard for them to get used to something new. Maybe because if you haven't experienced any alternatives it is very hard to imagine that something could be better.

I think bazaar's like the Turkish one referenced above (with guilds and standards of quality) are the best of all worlds. but to get there, a guild needs a strong, opinionated leader. Torvalds perhaps. (PHK seems to fit the bill too :-)

So, what I suggest is the issue here, is too many people trying to build a cathedral in a bazaar - they want everyone to build it their way but cannot persuade enough others to follow them. As such the bazaar is full of many half-finished carpets instead of less, perfect carpets made by engaged teams.

Software needs many talented people, working together, mentoring new ones in "their way". A bazaar is great for getting different talented people to find each other and choose between many different projects. A Cathedral is great for encouraging apprenticeships, the promulgation of culture and so on.

I think that one of the best lessons to take away from this whole flame fest is that history is something to learn from, and not just blindly revere. But first, that requires knowing about history, which far too many people don't. The whole issue of whether cathedrals or bazaars produce better software is orthogonal, and needs to be addressed to different situations.

First, are there still places where cathedral style development is a very good idea, hinging on being necessary? Yes, but it has to be done right, and unfortunately, cathedrals aren't (usually) very flexible. The Mars rovers were all cathedral programmed and work pretty darn well; on the other end of the spectrum you can find all sorts of big projects that went awry, even with "coherent design".

Second, is cathedral design necessary for all software today? No. I'm fortunate enough to have the perspective of a second generation programmer; my father started in middle school by sending punch cards to the university and would get his results a week later. Careful planning was not just a good idea then; it was absolutely necessary. These days he has a computer with more power and connectivity in his pocket than was available when he was born. His turnaround time for seeing results on software changes these days is well under five minutes.

So the computing industry has come so far so fast that it's not just unnecessary to make grandiose plans in advance, it may actually be a bad idea in a number of cases (what was it Paul Graham said? "If you want a recipe for a startup that's going to die, here it is: a couple of founders who have some great idea they know everyone is going to love, and that's what they're going to build, no matter what.").

It's kind of nice that anyone can just pick up a computer and start writing software these days. Sure, sometimes it's scary too, to think of all the inefficiencies, and wrong results and security holes that plague software. But it's so nice and liberating to think: "you know what? I can do it better, I have the freedom to try to make something better." And you can develop all cathedral style if you want!

Could software be improved? Definitely! Should people learn from history? Yes, including the mistakes. Would more design and coherent vision help make better software? Maybe; go ahead and prove it (a quick note: unfortunately, the market doesn't select the best software or designs, so you will have to prove it some other way, but being successful market wise is a good start).

I will tell you, though, that Snow Leopard missed its ship date by 8 months (of course, you never heard about that because Apple is smarter than to announce an OS ship date before it goes GM). It was also nearly un-usable for about a month, and only marginally usable for two months beyond that, during development. Lion suffered from none of these issues...

I guess post Snow Leopard era OSX and Apple apps just feel cheap, unpolished and rushed. It actually feels a lot like Windows, where quality of different parts of the system varies wildly.

Examples: Finder and anything file browsing related beachballs much more than before (particularly when dealing with network shares). Visual design seems to have taken a hit - everything seems to just consist of either different murky shades of grey, or jarring skeumorphism. The 10.6-era AHCI kernel panic hasn't happened for a while now, so maybe that was fixed in a newer revision of 10.7 or 10.8, but userspace stability seems much worse. Some of it is crashes, some just weird UI glitches that require a restart or Force Quit. Old problems (like Mail.app reliability or the heap of shit that is iTunes) are ignored completely in favour of sexing up the UI. (the runaway dynamic pager issue FINALLY seems better on 10.8, but again we get loads of other regressions)

Add to that what you say: the "improvements" leave most people I know completely cold or even get in the way, and that's not just power users/developers. Since we're "paying" for the (subjectively) unnecessary bling with regressions, the overall impression is that of a negative change.

Not that any of this is particularly relevant to the development model. You can end up with festering layers of crap with either model if nobody feels responsible for overall quality. I suspect the objective for Lion was "make OSX look kind of like iOS and get rid of all of that GPLv3 software" and that's basically what was achieved. I guess bug-free software isn't good business, Microsoft were printing money for decades.

You can say that again. Frankly, after using Mac for 10 years, this is my last one.

Hackintosh or something even worse (and I am with you 100% on the horrible state of OS X) -- I wouldn't want to discount the quality of the hardware.

In fact, it actually reminds me a lot of the change from FreeBSD 4.x to 5.x ... things worked, people were happy ... and then boom.

That was when all of my desktops and laptops stopped running FreeBSD and I "switched" to the mac. I am fairly certain that once running SL stops being practical, I will switch again.

Nothing against a good conspiracy theory, but if you truly think this is intentionally planned then this thought is a bit … dumb.

It does feel that Apple's development model is changing to an annual release cycle, wherein larger/systemic bug fixes are deferred to the next major release, rather than in a point release. In fact, most OS X release these days feel like "minor tweaks that make consumers happy + a truckload of bug fixes and plumbing". The fact that they deferred so much 10.0-era plumbing to 10.6 and beyond is indicative of why there's a lot of regressions, IMO.

Minor irritations like reworked Spaces and full screen mode making a second monitor pointless are irritations, but I wouldn't regard them as reflective of the quality of Lion and ML.

Whether products are developed according to what users want vs what builders want has almost nothing to do with bazaar vs cathedral.

Indeed, many bazaar-style projects (the linux ecosystem in general, gimp, etc) are almost cautionary tales about "builders creating what they want for their own niche purposes and if anyone disagrees they're free to fork off".