I've been running a reasonably large kubernetes cluster (~600 cores) in production for the last 6 months or so. I've had to deal with hotspots, both IO, CPU and load (which could be either). Getting requests and limits dialed has been a bit of a challenge.
I worked with Mesos a bit last year and miss the sophistication of the scheduler. Granted, I never ran it at the same scale, but I felt like the approach was much easier to grok.
I've seen a few projects floating around meant help dynamically tune resource utilization in k8s. By this time next year, I imagine those will be fairly commonplace. I'm definitely looking forward to using descheduler:
Essentially, yeah. You don't really deploy a framework, you just start its scheduler. I recommend the technical paper, it's clear and short[0].
Weirdly enough, Mesos resembles a system I was building in my head that I thought could compete with Kubernetes... Taking the resource supplier (agents in mesos-speak) and consumer (frameworks in mesos-speak) paradigm to the extreme.
[0]: https://mesos.apache.org/documentation/latest/architecture/