I wanted something like this 4 years ago so I started writing a project called Drop [0] with the intention of making it possible to add custom visualizations through a plugin interface, providing plugin developers with tools such as beat detection, and a decent user interface. I added a lot of features and basically finished the interface but never got around to finishing the plugin portion.

[0] https://github.com/xNul/drop

You might be interested in this btw

https://delu.medium.com/a-perceptually-meaningful-audio-visu...