It's probably going to cost you time, more than hardware, I'd honestly suspect? Any simple Atom-based NAS should really have enough "oomph" for like, a REST HTTP API that you can talk with. I have a 4x core C2000 about a foot from me that would do great. You can piece one together for a cheap few hundred $ (like $300 can probably get you something decent). But you're going to have to program all the shit together, that's consuming.

I've been looking at this for my home. Realistically I've just been spending a lot of time so far just modeling my database schemas so I can eventually put APIs on top of it all to talk with.

Mostly just make sure you pick some devices you can think you can interface with, have been reverse engineered, or get ready to reverse it yourself I suppose (which might be easier than you think, considering how bad embedded security is)...

I did something similar (not a router, but we considered it), and its not that difficult on the hardware side. Our home automation hub was just a Raspberry Pi 3, and had more than enough power.

If I were doing this just for my own home I'd probably get a half decent NAS capable of running Linux and then run home-assistant.io on it rather than building everything from scratch. Home Assistant gives you a decent UI, integration with almost any hardware you can think of, programmable automation, and an API to build against if that isn't enough.

For the router you could look into PFSense. I absolutely love it. Use it both professionally (as a firewall/VPN endpoint/access filter/upstream load balancer) and for home setups.

On the NAS side there is FreeNAS, however I don't know the two would interact or if you could directly install both packages into one server (I don't see why not though, even though PFSense probably wouldn't recommend it).

About the same as a c2000 nas