Many other replies speak of the difference between “GPU” and “CPU” without describing it—it’s thread divergence. Processing stream-like data (network streams; compilers and other parsers; human input devices) can branch often and unpredictably using normal algorithms. Think for example how you would tokenize JSON without branching. A single GPU core is quite slow for the general case anyway. They are mostly good at math and bitwise operations. They were able to get this way because of the assumption that there would not be thread divergence for the primary workload, i.e. matrix math on contiguous blocks of memory, which harkens back to their origin as actual Graphics Processing Units.
If thread divergence weren’t such a killer for branching programs, you could probably write anything in CUDA. It’s not very limited; you can write most anything that can be expressed in non-exotic C. A lot the work has to do with schlepping textures around, which is an artifact of the expectation that your code will run at a distance from the messy, branching, unpredictable main program state, so everything (textures, shaders, etc.) should be loaded, processed, and ready to go when it’s frame-buffer time.
Thread divergence.
CUDA-cores are linked and take if-statements and for-loops together. This means that if one CUDA core takes a for-loop 1-million times, 31-other CUDA cores will take the 1-million loop with them. (32-cores per NVidia SM). EDIT: CUDA keeps things semantically correct per thread by "throwing away work" (execution mask is disabled, so those 31-other threads effectively execute NOPs), but the cores are wasted.
Worst case scenario, your CUDA cores accomplish 1/32th the work they could do, as they spin idle waiting for the last 1 thread to complete a for loop or its unique combination of if-statements.
Matrix multiplication doesn't have any thread divergence, because all threads loop the same amount of times on all workitems. This means that Tensors / Deep Learning simply ignores the problem, because they're written as matrix multiplication problems.
Avoiding thread-divergence, or mitigating its effects, is possible, but requires advanced programming skill that few have.
--------
Examples:
Chess -- Traditional multithreaded chess algorithms have every thread check out a different branch of the chess search tree. But because each branch has different positions and moves, its very difficult to code it in such a way that all threads are doing useful work.
Web Servers -- If every thread is handling a different request, then every thread probably traverses a different set of if-statements and loops. This is a thread-divergence nightmare.
Databases -- SQL Databases traditionally have each thread running a different, independent SQL query. Like web-servers, its difficult to imagine parallelism from a high-level.
-------
However, elements of Chess, Web-servers, and Databases can probably be efficiently parallelized onto GPUs.
Chess -- This github repo demonstrates that enumerating positions can be done GPU-parallel: https://github.com/ankan-ban/perft_gpu
Web-servers -- Text parsing, Regex, and many other problems common to Web-servers have been parallelized on GPUs. The main question for me, is whether or not the PCIe traversal would be worthwhile (lower bandwidth than RAM, high latency). Web-servers tend to be I/O heavy and less compute heavy... probably not dense enough to benefit from GPU compute.
Databases -- Merge-join and Hash-join probably can be parallelized to GPUs.
Figuring out the optimal CPU + GPU team is going to be a research problem over the next 10, maybe 20 years.