I used to joke developers should get workstations with SPARC Niagaras or Xeon Phis for that reason: core count is going up and a CPU with a dozen small cores is much cheaper to build than one with 4 beefy ones. Now some of the low-end chips Intel is pushing have 4 SMT2 cores. HEDT is on the 16 SMT2 core range and, if our software continues to be single threaded, there will be a lot of silicon being used for nothing more than spreading heat.
The incentive to do it when most computers have two SMT2 cores is small, but as the average device starts getting 4 or 8 SMT2 cores, or 8 asymmetric cores, the incentives to make things run faster in parallel get better and better.
Consider this: We mostly parallelize compilation today at the unit of a single compilation unit.
We do this for two reasons: It's simple, for languages where separate compilation is possible, and it's good enough when you have a small-ish (smaller than the number of compilation units) number of fast cores.
But a lot of compilation can be reduced into far more fine-grained parallel steps.
E.g. parsing can be turned into a chain of array operations along the line of finding possible start/ends of tokens, finding possible starts and ends of quoted segments, finding possible starting points for matches against various rules based on the possible token starts etc. You can further build syntax trees as arrays of vectors. If you have many enough cores it may pay off to subdivide the work to extreme levels like that.
But it's hard to reason about. We're not used to it, and most of us are no good at conceptualising it. Some are.
For a real-life example of a compiler that actually runs on GPU's and aims for massive parallelisation (it compiles a subset of Dyalog APL), check out "co-dfn" by Aaron Hsu. It's very terse and complex to read, and frankly I've only spent time looking at it very superficially, but it's an amazing piece of work. Now, if it was easier for mere mortals to actually write code like that, and we had a more compelling reason (many, but slower cores), we'd probably see a lot more parallelisation. In the short run it'd be extremely painful, but in the long run it might well benefit us.
When programming the level of friction is from annoying to high unbearable, depending on the programming language. It should be trivial to implement these solutions, as trivial as the sequential solutions.
It's not. Until this is the case in mainstream programming languages, I doubt we'll have CPUs with 1000 cores in our smartphones or laptops. I mean, we'll have them but they'll be useless because 99% of software today is practically single-threaded. The reason multiple cores are good for laptops, for example, is because we're multi-tasking between multiple applications, not because those cores are frequently used by day-to-day apps. I'm talking about random small apps, not compute intensive ones that have to use multi-threading intensively (rendering, compiling, what have you, professional level software).