It is very gentle indeed.

Personally i've come to conclusion that computers are better at managing threads than i am, just like compilers are, for many decades now, better at managing CPU registers.

It can definitely be enjoyable to ponder on some threading puzzle, now and then, but I very much prefer higher level abstractions like CSP or STM. Threads are not for human consumption.

>it is not 100% guaranteed that threads will perform their operations truly in parallel, that is at the same time: it really depends on the underlying hardware.

I thought it really depended on a lot of factors, most predominantly thread scheduling (based on thread priority)[0]?

[0] - https://docs.microsoft.com/en-us/windows/desktop/ProcThread/...

It depends on a lot of factors, but if your CPU physically does not have multiple cores then you can be 100% sure that your threads will not be executing code literally in parallel.

I think on most operating systems, if you have a multicore CPU and very little load apart from your program, and you run CPU-bound code in multiple threads, then you will see those threads executing in parallel in different cores.

To build on what you're saying: while true from the user perspective, out-of-order execution and instruction level parallelism means there are some types of parallelism happening at the single core level. However, these systems are designed to produce results to the end user as though no micro-level parallelism is occurring.

I mention this mainly because I've recently discovered the joy of SIMD intrinsics. While it's pretty difficult to gain anything from out-of-order execution (cough other than a spectre) it's possible to take advantage of SIMD through compiler autovectorization, intrinsics, or assembly coding. SIMD doesn't have the problem of race conditions though as the parallel operations are non-conflicting and the user view of the computation is synchronous. I imagine under the hood there are all kinds of tricks at play for complex instructions that involve different microcircuits that take different numbers of cycles.

EDIT: clarified last sentence

SMT (Like Intel's hyperthreading) can actually run 2 different instructions from 2 different processes in the same clock cycle on a single core.

Are there any operating systems that let you explicitly run CPUs in parallel?

I'm thinking n threads where the process requests the operating system scheduler to start and stop them all at the same time. Of course the OS would be permitted to refuse if it wasn't capable (or just not allowed).

A synchronised cpu master-slave relationship could be beneficial to parallelize some of the middle ground between instruction level parallelism and multi-threading

You can do a set affinity on Linux but it isn't a guarantee.

Why would you want to override the scheduler like that? That would also mean not scheduling your process until all the CPUs were clear, and greatly increasing scheduler coordination overhead. So it would decrease overall work throughput.

CPU affinity is genuinely useful, but I can't see why you'd want this kind of temporal affinity. Especially since you could have different CPUs running at different speeds!

>Are there any operating systems that let you explicitly run CPUs in parallel?

That's a good question. I don't have the answer, to be sure, but I would think part of the problem is conflicting priorities and who would "win" in that scenario.

CUDAs in a GPU might be more worthwhile for that approach, maybe, but I could also be talking out of my arse, here...

You can get that effect by pinning processors to threads and you can do the sync yourself with a barrier (like Java CyclicBarrier, not a memory barrier.)

Please note that

  Application can allocate memory. 
  Application cannot allocate CPU; 
  OS does that;

Is there a good overview of different concurrency/parallelism models and related abstractions somewhere?

Seven Concurrency Models in Seven Weeks: https://pragprog.com/book/pb7con/seven-concurrency-models-in...