I'm reminded of the old adage "I'm writing you a long letter, because I don't have time to write a short one."

(Often attributed to Mark Twain, but similar sentiments were expressed by many before him.)

I have two projects that I consider to have nearly-perfect code. Both are their third iterations, and I think they're stable at this point.

Kinda goes: 1) Make a bad solution exploring the problem 2) Explore a good idea for how to solve the now-understood problem 3) Mature the good idea through usage.

Do you use TDD? I'm not religious about it in general, but when I'm lost, confused, and easily distracted, I start with TDD to write the dumbest possible code.

No.

It's really an issue of not being sure at first what needs to be flexible & data-driven vs handled in code. If make everything data driven, then it becomes this horrible mess where your input is basically a program and your actual code ends up being a terrible interpreter.

I tend to just build things bottom-up, and start with a small bit of functionality, then when I have enough small bits, I bolt them together and decide what I need to abstract at that point, do refactoring on the smaller bits and provide data to them from the caller. Then repeat that continuously until I have all of the functionality I need.

It might be different for other people, but I need to have working code before I can it abstract.

I don't understand. Tdd gets you to working code (if on a subset of all your data) extremely fast. There are both bottom-up and top-down styles, neither of which is particularly wrong.

The TDD aspect is not really here nor there.

To provide a more concrete example, say I have a function that performs a transformation on a piece data. From what I currently know about the data, I can parse it using a regex. So I code up the function that accepts data, it runs the regex and provides the transformed result. Great.

Now as I'm continuing my work, I notice that some other data requires a similar, but not exactly the same transformation. It can be done with a slightly different regex. So rather than duplicating functionality, I modify the transform function above to take a regex as input along with the data. Everything works as expected.

I get further along in the project and I realize another piece of data needs a somewhat similar transformation, but this time it's just slightly too complicated for a stand-alone regex, it needs to be a function.

The "smart code" way to handle this would be to create another transformation function and call that instead. The "dumb code" way of handling this is to generalize the transformation function such that I can pass in some descriptor for the transformation, and have the transform function return the correct result.

That's the crux of the issue. I rarely have enough information at the time of writing to know just how generalized to make a function. If I created this hyper-generalized transformation at the beginning, but never needed anything beyond the original simple regex, I would have wasted a bunch of time creating code that's needlessly confusing.

TDD is perfectly applicable for development, and would help tremendously with the refactoring aspect, but what it doesn't help with is information that you don't yet know about.

TDD is good for features (like web apps) but not so much for algorithms.

The difference is that you only need to support a tiny fraction of possible features / use cases, but your algorithms need to be correct for a wide range of inputs.

I disagree. You code your algo for one input, come up with a corner case, write a failing test, refactor, repeat.

For an algo, let's say it operates on a list, I'll start with test f([]) == 0, and implement f to output the constant 0.

And then go from there.

That only works if the algorithm isn't already understood and so you don't know what optimal is. If you tdd sort there is no way to get divide the list to small chunks, insert sort each small chunk, then merge sort them. (20 years ago this is what gnu sort did, I assume that this algorithm has changed now that cache is a bigger factor in performance). Even knowing that the above is your end goal TDD to get there is wrong because you don't know how the best algorithm will change when something else comes up as important.

Tests are good, but they need to be universal for any implementation, which means you often cannot tests the internal details that prove you didn't use bogo sort (picking a pathological example to make the point)

Ok but honestly, most people are not coding raw low level data structure algorithms. They are coding business algorithms. "Here's a table of accounts, calculate what they owe"

When one says "wide range of inputs", as gp did, one is not talking about low-level algorithms; one is talking about a business algorithm. You should be using TDD for this.

Why does it have to be this polarized - stupid code & smart objects. Writing smart code that uses smart objects is better than stupid code. Writing stupid code that uses smart objects is indeed difficult.