I have had the same experience and instead of Pandas have been using numpy-groupies to handle aggregate/groupby operations. It's quite performant and feels a bit cleaner to use than importing pandas for a couple operations.

https://github.com/ml31415/numpy-groupies