> so how do you know which model to use?

That's like asking, "How do you know what algorithm to use?" You can come up with a justification for why a particular algorithm may be suitable for your purposes, but you'd be hard-pressed to explain how you go about choosing an algorithm. It's the same way with rules for a codebase... you can justify why you are using a specific set of rules, but explaining the decision process that led you to choose those rules is much more difficult.

That's why tech leads don't grow on trees.

I'll add to what the original comment (about "confused concurrency") was saying, because it matches with my experiences. The good lead developers are the ones who are able to get the job done (design systems) and keep the concurrency model simple. It's often possible to keep concurrency mostly simple and mostly safe.

I worked on a large extremely high performance system (dozens of developers) that’s almost entirely lock free, and constantly mutates shared state. Concurrency issues weren’t really a big problem in that code base. Developers took a long time to ramp up, but that’s an OK tradeoff in my book.

None of this would have been possible without lead developers that designed and continuously refined the guard rails that our internal “standard library” implemented.

Oddly, I now find it easier to reason about lock free algorithms than mutex based code (and I wrote the latter for 15 years before working on the lock free system)

>Oddly, I now find it easier to reason about lock free algorithms than mutex based code

That is because lock free code composes. Mutex based code does not compose. With lock free code, I can freely call another function, knowing I cannot deadlock. With mutex based code, I have to be really careful about the state of my mutexes and think if the code I am calling will attempt to lock one of the mutexes I am holding, thereby causing a deadlock.

lock-free code is free of deadlocks, but it is not free of livelocks and usually there's no guarantee of progress...

In mutex-based code, usually you want only open calls (i.e. calls with no mutex held, so you only ever hold one mutex at a time).

> usually there's no guarantee of progress

I think lock free guarantees that at least 1 thread will make progress.

https://en.wikipedia.org/wiki/Non-blocking_algorithm#Lock-fr...

thanks for the correction. I thought even with that guarantee, there is no bound on progress (so progress can in fact be very slow). lock-free algorithms are not used much because of the slowness, AFAIK.

Lock free means that at least one thread is guaranteed to make progress. You might be confusing this with atomics, which could still be used in a spinlock and would then not be lock free.