It is helpful to store configuration with code, but you don't have to include secret values in your code. It's much better to use a purpose built service like credstash[0] to store secret values while you keep the name of the secret and (possibly) version in a repo with your code.

[0] https://github.com/fugue/credstash