The composition problem is considered the hard problem of panpsychism. But as your intuition correctly points out, it is definitely easier than the original hard problem of materialism, which is a category jump from quantities to qualities.

The main difference is that physics is quantifiable, so we can come up with in-principle explanations of really complex stuff (weather forecasts, life itself, etc). With qualities it's a different ballgame, because we are talking about different subjects combining into one. Not saying that it's a dead end, but we haven't even yet started to define the problem, let alone solving it.

There are other more compelling roads. Rather than a combination mechanism, we can look at the problem from a decomposition perspective. This is easier because it's a really well understood phenomenon that we can actually study: the compartmentalization of one unified mind into separate minds, also known as multiple personality disorder or dissociative identity disorder. Here, we have a clear in-principle path to walk.