If you with the commercial router route, checkout OpenWRT / DD-WRT / Tomato. There are several forks, so depending on what features you want, and what hardware you have, you may opt for a different version. If you go with an existing x86/amd64, just install your favorite distro and start reading up on iptables configuration and management. While iptables to be quite powerful in terms of features, I find its' syntax is painful. There are several projets aiming to simplify this, by generating rules from another, simpler, DSL but it adds complexity. This is why I prefer PF, but that requires you to install OpenBSD instead of Linux. If you know your way around UNIX, and are ready to read up the FAQ and man pages, this shouldn't be a problem :).

Openwrt https://openwrt.org/ is where most of the docs are, though there are other similar projects. The most supportive vendor is probablr Netgear now http://www.myopenrouter.com/