I suggest you look at how people perceive your value. They obviously see you as being more valuable as being a manager than an IC.

I would look at how your 'brand' comes over and work on making sure people see the value you bring as an expert IC outweighs that of being a manager.

The more you increase your value as an IC the more costly it becomes to use you as a manager. Then you can look for a job that understands that.

I hope this helps