HBM's latency characteristics are about the same as any other DRAM however (including DDR4 or LPDDR4x).

So you have a high-bandwidth, but no better latency, L4 chip. And that's kind of hard to deal with and optimized against.

Xeon Phi had a DDR4 + HMC (where HMC was a early competitor to HBM), and getting good performance out of that was unusual. DDR4 ended up having slightly better latency, so the HMC cache would hurt some workloads.