How is that for a mixed metaphor?

That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be able to deliver an estimate, and the best approach to take is to provide an estimate with Quantified Uncertainty. The phrase is important, because the estimate is for managers and managers are all about quantifying risks, and uncertainty in a development estimate is definitely a risk.

When you say "I don't know how long this will take, because I don't know how much I don't know about the solution" you're giving an estimate with uncertainty, but you're not quantifying the uncertainty. It's got no bounds, it's limitless. Managers can't make any decisions with that, so it's useless information. But if you say "This will take 4 to 8 effort weeks, and I'll be able to give a more precise estimate after 2 effort weeks of investigation" that gives your manager something to work with. You're still uncertain, but you've quantified the bounds on the uncertainty, and decisions can be made. For example, solving the problem might not be worthwhile unless it happens within the next two weeks (eg: to meet a release deadline) so the manager can just delay the work until later.

The trick with this approach is setting the bounds. The man who taught me this style of estimating called it a Surprise Range. The lower bound should be an estimate such that you'd be surprised if it took less effort than that, and the upper bound should be an estimate such that you'd be surprised if it took more effort than that. Given whatever information you have about the requirements, you just keep pushing your bounds until you can honestly say "I really don't think it would take less/more effort than that."

As you get more experienced, you'll naturally start accounting for the unknown unknowns, because your past experiences with them will be nagging in the back of your mind while you're trying to come up with an estimate for some new task. This is especially true if you're estimating work on a project you've worked on before; you'll have a feel for where the trouble spots are and whether or not the task in question is going to get into those spots.

The other trick with this approach is the "more precise estimate after some investigation" bit of the statement. You need to provide some estimate now, buy some investigation time, and commit to another estimate later. Most of the time the second estimate will fall somewhere within the bounds of the first one, but not always, and your manager should be aware of that possibility. But with experience, you'll usually wind up within the original bounds and narrower than the original bounds too, which is quantifiably less uncertainty than before so you can show progress.

A month later my peer, a senior Engineer with the Company, thought "We should run 32-bit! How long can it take?", checked out the source and started changing files. Gradually people noticed what he was doing, resources got added under the table, finally a year later he had something working. Lots of kudos, smiles, what a hero!

I showed him the report I filed a year ago - and he said "yeah, that's about what I had to do. I'm glad I didn't see that to begin with, I would never have started".

Also, "two man-years" isn't a complete estimate using my approach. "One to two man-years" would be a complete estimate, providing a range. Sometimes you really do come in near the bottom of the range, and maybe a month into it if you re-estimated you would have come up with a lower range.

This brings up another point of contention between development and project management. When developers say "estimate", they mean "best guess in the face of uncertainty", but project managers tend think of developer estimates the same as plumber, electrician, and car mechanic estimates. Those guys are saying "I'm not sure exactly how much work I'm going to have to do, but if you pay me $X I'll do it." They inflate $X so that most of the time they wind up overcharging in order to cover the occasions when they wind up doing much more work than expected. The $X is a commitment, and the estimate is how much work is needed.

It doesn't work this way in software development, because there are factors out of the developers control. The developer can say how much effort a task is likely to take (the ranged estimate) but not how much time (the commitment.) The time depends on other things the developer has to work on, both expected and unexpected, dependencies on other developers and tasks, holidays and days off for the developer and others the developer is dependent upon, etc. That's all stuff the project manager is supposed to keep track of, so the developer doesn't have a full picture. Unfortunately, most project managers don't recognize this, so they treat the effort estimate as a time commitment, set an arbitrary start date, and expect the developer to be done X days later.

I don't, in large part because this shouldn't be a complicated problem to solve.

Your confidence probably serves you well (let me guess, early 20s?), but in this case you literally don't know what you're talking about.

But that's from personal observation, and I haven't made a point of recording my findings for analysis. I suspect you haven't either.

No.

> but in this case you literally don't know what you're talking about.

I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process.

FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least according to Alexa). I'm not speaking from inexperience: if this problem is hard for 37signals, it's because of failures of the implementation, not because the problem is intrinsically hard. Engineers who don't recognize that are not ones whose competence I would put trust in.

The problem is easy. Seriously, it really is. Most of the programmers here have probably done something almost exactly like it ten or more times in their relatively short careers; I know I have. There is no real intrinsic difficulty to the problem.

I'm not saying that the programmers are 37signals were incompetent. I'm saying that I have do not trust that they are competent. I have no reason, especially in the face of this evidence, to believe that they're competent. They made a successful software product: so have lots of other programmers of questionable competency.

The fact remains that at least in this example, one of their programmers points at an intrinsically easy problem and says, "This is really hard because I'd have to do a lot of stuff that I really shouldn't have to do" except he doesn't seem to realize that he shouldn't have do those things. So tell me, in face of that obvious fact, why should I trust 37signals' engineers' competence?

It's easy in the simplified mental model that you have constructed devoid of real world context. Any engineer worth their salt knows the devil's in the details. I'm going to stop arguing now because you're not even responding to my actual argument. It's not a zen thing, you should be able to get it if you actually read my comments, but just in case you need a koan, ponder this fact that I would bet my life savings on:

There exists a potential feature which could be implemented in Basecamp faster than in your product (and vice versa).

I constructed my mental model based on the detailed description by a BaseCamp engineer of what he'd have to do in order to solve the problem in his system!

> I'm going to stop arguing now because you're not even responding to my actual argument.

Welcome to my world: you've been arguing this whole time as if I were claiming that BaseCamp engineers were incompetent when I've only been saying that I don't trust that they are competent.

> There exists a potential feature which could be implemented in Basecamp faster than in your product (and vice versa).

There are probably many such features. That's really beside the point, unless you're claiming some additional knowledge here that the decisions made in this particular design contributed to the ease of those potential features. If you're only assuming that because you trust the competence of 37signals, your entire argument is circular and depends on the very claim I'm contesting.

Do you realize how weaselly that is? What does that mean? Connotatively it takes a huge swipe at the 37s team without actually making any commitment. You might as well have said nothing at all if you're not going to take a real stand.

There's a distinct difference between saying that you know someone is incompetent, and saying that you don't know that someone is competent. It's the same as the difference between agnosticism and atheism.

What I am also saying, and what you're probably really arguing with, is that the linked article constitutes evidence against 37s' competence. That's still very different than saying that it shows that they're incompetent, which is how you've been mischaracterizing my posts.

Understood, but it's still weaselly because you're offhandedly casting aspersions on a group of people's ability ("I have no reason to believe those people are competent") and then pretending like it doesn't hurt their reputation because of the exact words you spoke ("I never said they were incompetent, just that I have no reason to believe they are competent").

Agnosticism vs atheism, while logically the same distinction does not have this libelous aspect to it.

Why the h... doesn't their ORM, or even better their database, already take care of cascading updates?

Why does it have to be implemented manually for every code that updates something? (... which is of course expensive and prone to errors)

So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.

The irony, of course, is that a single Postgres server is frequently (IME) more reliable than multiple MySQL servers even when the latter is setup for HA.

Also note that if you want to trade some of the ACID properties for better performance and replication, the so-called NoSQL databases (CouchDB etc) seem to be a better trade-off than, say, MySQL with MyISAM instead of InnoDB.

While I fully agree that trade offs like yours do occur in our profession, their mere existence is no argument against my initial non-trust of 37signals' competence: you need to actually demonstrate that this problem is one of those cases, and I think that would be difficult to do.

The problem of associating entities in a way that their associations can be modified easily and consistently (i.e., normalization) is effectively a solved problem. Just like the programmer complaining that his program is slow because he's doing a linear search through a linked list, a BaseCamp programmer complain that a requested feature is difficult to implement because their data is in an abnormal form does not inspire confidence, and certainly gives no reason to trust his competence.

In Erlang, the most common data structure is the linked list. There's no way to search them other than linearly. You are claiming that in general we should suspect that Erlang programmers are incompetent?

In the case of sharding by project, it actually seems like a decent move given that a single Basecamp account has potentially large space requirements (75 GB), and you can't realistically provision enough space on a box to meet all the needs of a bunch of expanding accounts on that box. Being able to throw an entire project onto a new server and assuming that everything associated with a project is on that one server are both nice simplifying assumptions which seem like they would strike a nice balance.

Please stop painting everything with broad black and white strokes, programming and business in general are frequently more about striking the right balance of compromises than about knowing the "best" way to do things. There's usually not a universal "best". 37Signals seem to be doing a marvelous job balancing those compromises between their various disciplines, especially given their massive success with customers.

Edit: This is assuming they're storing attachment files on the same machine, which might very well be wrong. It would be pretty hard for one group to get up to 75GB of todo's and other text content...