I think this really comes back to the question of how custom your technology needs are. My local coffee shop has a website and apps but the owner doesn't need to be a programmer because they have the _exact_ same needs as everyone else, and something like ToastTab can be licensed and used without modification. The same is true of a lot of online stores where their business is conceptually very similar to many others.

If your business is less like that, however, I do think it's critical to find someone you have a high degree of trust will make reasonable calls. Once you start needing things like custom development you're saying that technology is core to your business. That doesn't mean the CEO needs to be a techie any more than the CEO of a restaurant chain needs to be a chef but they definitely need someone whose judgement the CEO respects.