We also compile and read the template once to determine only the keys you care about from the template to determine what Consul services/keys to look for, so we're not requesting ANY changes.
confd also has dry run support: https://github.com/kelseyhightower/confd/blob/master/docs/no... It was apart of the initial release of confd.
These distributed K/V stores are designed to offer a minimal feature set, and the watch support is a perfect way to deliver events to clients. While HTTP call backs would be nice, I think you have to draw the line somewhere. Since these types of systems, consul or etcd, sit at a critical point in the infrastructure lets keep them lean and focused. But that's just my opinion.
https://github.com/kelseyhightower/confd/commit/16d6b347c3d3...
What really bugs me about these solutions -- confd, consul templates -- is that managing templated configuration files isn't as quick, but has been trivially possible with normal configuration management tools since their inception. When I think of managing HAProxy configurations, for example, I don't see Chef as a bottleneck. I can parallel SSH and run Chef on those hosts in 20 - 30 seconds. I don't care about the configuration being pristine, either.. because that's what backend health checks are for. That extends to a lot of their examples, but it's one of the ones that stuck out to me.
The things that I think are hard, or harder, are small bits of data, like database credentials, etc.... where people absolutely don't want to be making an HTTP call for every single thing, or don't want to define a template for hundreds of individual files that might be needed. That's a real bottleneck - you need something on top of these solutions to manage that. So for every file you care about being updated fast... you still need to manage a corresponding file.
I don't know necessarily how to solve that -- I treat K/V pairs as file name/contents pairs in my tool and don't think twice -- but it definitely feels a bit wrong.
What would be really neat is some way of having consul initiate an event instead of having pyconfd (or confd for that matter) sleep and check every n seconds. This may already be implemented in consul upstream but I haven't been paying very close attention...
Edit: dry run is very neat :-)