You could run this in a single query `count(*) .. group by (tag, category_id)` and then post process the results.

counting by groups is still relatively expensive though in postgres; I built an implementation which uses estimate counting (via query planner) for larger buckets, and exact counting for only smaller buckets, but my use case allowed for some degree of inconsistency.