sauceop

I feel like a lot of semver purists have never maintained software with a lot of downstream dependencies. The gray area about what is a breaking change is huge and it's hard to usefully improve software without making changes that are breaking under some definitions. The broadest is that, if a downstream program was working against version x.y, then it should always work unmodified against x.z.

One obvious case is bug compatibility and undocumented behaviour - it isn't that uncommon for software to depend on bugs or undocumented/unintentional behaviour. But if you're exposing a language like SQL, it's even messier. E.g. if cockroach added a new feature to the SQL dialect, then it could break downstream software that was assuming a particular SQL string would return an error. Or if you extend existing functionality, say supporting a wider range of arguments in some function.

There's also unspecified behaviour and implementation details - e.g. if a query without an "order by" happened to reliably return rows in a particular order before an upgrade, but it changes after an upgrade because of an algorithmic change, is it a breaking change? Common sense would say no, because it's not documented behaviour, but pedants could disagree.

And of course there's performance changes - if a query runs 100x slower after an upgrade, but returns the same result, is that a breaking change?

If you take the most pedantic interpretation of semver, you either have to bump the major version number all the time or you are very limited in how you can improve the software.

Agreed, I think the issues you're touching on are fundamentally difficult problems that don't have a single solution. For example.

* storing complex data and supporting multiple access patterns efficiently (like joins) is hard and involves trade-offs (e.g. normalisation vs denormalisation, benefit vs overhead of indices).

* data consistency is hard, particularly once you are at a scale that requires distributed storage

* durability and disaster recovery is hard

* failover and availability, particularly for a stateful system, is hard

The right solution is completely application dependent. Usually the best approach is to avoid hard problems to the extent your application allows it, e.g. by accepting relaxed consistency, partitioning the data, or limiting the use cases the system supports. Then you can outsource the hard problems that remain to an existing solution (e.g. a transactional database).

Microservices help address some of those problems, but you need to recognize when they don't (e.g. you need strong consistency across multiple services) and adjust your designs accordingly

The open source projects that are most threatened like this are the more complex infrastructure software that doesn't naturally get build as a side-effect of other work.

If you look at a company like Red Hat, they're focused on building things that will work reliably for the diverse needs of their thousands of paying customers. Whereas a lot of open-source contributions are really about solving the contributor's specific problem.

Sometimes if you get many people each making a contribution that scratches their specific itch you end up with something great. Other times it isn't cohesive.

If you're running IT for a major corporation and you have an issue with your OS that prevents your business from operating, what do you do? Hope that your IT team figures out an issue they've never seen before in code they have no particular expertise in? No, you want to be able to pull in people who actually know the technology and have seen it all before.

It's a free-rider problem - if large cloud companies fork every successful open source infrastructure project, don't contribute back, and pull away a significant chunk of users, then there's a lot less incentive for other companies to invest in future open source development, because they'll have fewer users and get fewer code contributions.

Potentially that shifts things from a good equilibrium where everyone reaps the benefit of many people contributing to the same projects, to a bad equilibrium where every major cloud player develops closed-source products separately.

I don't think it affects all open-source projects, particularly not small ones or ones where some is scratching their own it. But it's hard to build certain types of complex production-quality infrastructure without full-time employed developers working on it.

It depends on how important you think it is to optimize for future readers or maintainers of the code. I weigh that pretty heavily and a do think relatively minor style issues impose a tax once they're pervasive in a codebase.

Tooling is great, but I think there are a lot of style issues that aren't readily enforceable with linters - commenting, naming, control flow, abstraction, etc.

We had a senior engineer (who sadly moved on to a different company) periodically give a talk to new hires about good code review practices to set expectations and ground rules. Some of the key points:

* Authors are expected optimize for readability of code, even if it takes longer to get the code in.

* Reviewers should generally only start reviewing code if they are ready and willing to continue the review to completion (i.e. no drive-by reviews). Once they start reviewing they should try to minimize response time.

* Focus the reviews on testing, correctness, readability

* Consistent code style is important (see optimizing for readability)

* Break code reviews into the smallest reviewable units. Often these are somewhat large because parts of the change don't make sense in isolation

* The code author is responsible for writing code in a way that it convinces the reviewer that it is correct (see optimizing for readability)

* You should approach it collaboratively - you're working together to get the work done and make the codebase as maintainable as possible

* You should aim to leave each bit of the codebase that you touch in at least a good a state as it was previously

I don't have your context, but if the changelist author is writing multithreaded code where there's no way that a reader can convince themselves of the correctness, then my heuristic is that the burden is on the author to improve the code to be easier to reason about and add enough tests to exercise all of the interesting code paths. There are always techniques to simplify code or make it easier to grasp.

Consistency of code style does matter, up to a point, to people reading and understanding the code. As you suggest, setting up ground rules and tooling helps

Code authors who don't see the value in consistency of code are potentially a problem - if authors are submitting code reviews with hundreds of actual style issues, that's either a failure of process or the author to write readable code (not sure if that's what happened in your examples, to be clear).

The approach that I've seen work is to have high standards for code, but communicate the expectations upfront and make sure that code reviews are grounded in those standards (for new people, this means over-communicating, but it becomes more intuitive once you've worked together for a bit). And whatever you do, be constructive, not rude.

If your codebase is going to be maintained for a long time, letting sloppy or inconsistent code in is going to make it much harder to maintain or evolve in future, so just lowering standards to make people feel better doesn't work. It's totally reasonable to hold people to a high standard if they're submitting code that others will have to maintain.

A lot of the problem is if people start off with the expectation that their code should be merged with minimal modifications, they're going to be upset when a code reviewer seems to be moving the goalposts on them. Also if the reviewers comments are arbitrary (or seem arbitrary) because they're not obviously grounded in consistent standards.

I've seen a project go through the process of tightening up code review standards and it was initially painful (a lot of people used to the old way feel like they can't move fast enough, etc) but ultimately worked out and resulted in people shipping working features at higher velocity with far less time spent on rework and maintaining overly-complex and buggy code.

There's also way less conflict in onboarding new developers because standards have been made clear in advance and reviewers are expected to be respectful and coach newcomers through the review process.

The hard part, I've found, is working with developers who can't or won't get code to the expected standard, even with coaching and detailed feedback. As in, repeatedly making the same simple mistakes pointed out by reviewers and failing to understand and address relatively straightforward feedback. I'm not sure if there's a solution to this, beyond minimizing the damage and trying to move the person to a role that they're more suited to. In one such instance the person was, in retrospect, a complete liability - they switched to a different team with looser code review, got a "bugfix" committed with some obvious errors that, if not luckily caught by our team, could have potentially had disastrous consequences for some customers, then left the company. Maybe the lesson is that high code review standards contain the damage from such people.

Edit: I also think this requires code reviewers to approach the reviews with the attitude of "what do we have to do to get this merged without compromising our standards?", not a desire to tear down the person or block the code change.

Speaking as someone who makes a living contributing to open source, it's a frustrating trend that the major cloud providers (Amazon being the worst offender) have decided that it's in their interests to take open source projects and never contribute back or even interact with the community.

It's less clear to me whether this is rational on their part - whether the incentives have actually changed. It's generally been in companies' best interest to contribute back to open source projects they are building products around, because:

You need to contribute to projects to influence the direction of them.

And you will need to make changes or fixes to your version of the project, and the maintenance cost of an upstreamed patch is much lower than the cost of maintaining an ever-growing stack of custom patches.

I've seen companies before make the bet that they are better off forking a large project because they can "move faster". And it is true for the first year or two, but after a while the burden of keeping up with the backporting of changes from upstream takes multiple engineers spending large chunks of their time to keep up with.

In some cases you might be able to offset this just by hiring more engineers, but how many of the best engineers really want to work on backporting changes to a bizarro fork of an open source project? I suspect this is the bet the cloud providers are making - that they can grow the product fast enough to the point where they can throw money at it. We'll see how that works out for them.