yyparm

I tend to think the problem identified by the author is more of a fundamental problem with long-lived development branches that diverge with master - stuff will change under you and it's not always easy to notice.

It's also kind of a problem if underlying parts of the system that you depend on either are churning that much without that being communicated or if they have poorly-defined interfaces and are prone to accidentally breaking things that users depend on.

It's not entirely clear, but there may be some testing gaps too if bugs weren't found.

Maybe merge-based workflow helps a bit but it seems like the pain would still be there regardless.

This makes a lot of sense to me. If you're maintaining a large project having lots of non-cohesive commits make it much harder to figure out what the logical changes were. I care about the set of code changes that were required to add a feature or fix a bug. I don't care about the set of changes that were required to get halfway to working code.

This is also potentially a big deal if you're doing maintenance bugfix releases for a project - that's way easier if porting the bugfix to an older branch just requires cherry-picking a single commit.

Last time I looked at Presto, it works fine for simple queries (e.g. scanning data and aggregating it into a small result set) but the performance was prone to falling off a cliff as queries got moderately complex - it comes up with a bad query plan or query execution OOMs when data doesn't fit in memory.

Hive and other SQL-on-Hadoop systems tend to do better in that department.

One lesson to take from this is that it's often worth spending an extra half hour or hour validating that your assumptions about where time is being spent are true.

I've seen two examples recently where a potentially impactful optimisation was added to a product but it didn't actually work as intended because of minor errors. It took a couple of years before the performance bugs were found, which required several hours of work.

In the first, a O(n) algorithm was replaced with a O(log n) version, but part of it remained O(n) for a subtle reason. The code was still faster by a constant factor so it wasn't totally obvious. In that case validating that the algorithm was actually O(log n) by doing some experiments for larger values of n would have revealed runtime was increasing linearly.

In the second, an optional argument was added to a method that triggered an optimisation for a particular case, but it was never passed in at any callsites. In that case many different tests could have revealed that the optimisation wasn't effective or that the new code wasn't even running.