Completely agree. The PM is there to help the team figure out what to build, and help them succeed in building it. They can do this in a number of ways: Being an expert on the product is "must have", but they also need to know the market, the needs of users/customers, the metrics, and more.