What do you mean by this? Surely some things require some configuration. Don't they?
1) CS papers are normally about a single or logically-related set of concepts. If you can't reduce the code to a single file, the paper is probably too broad or you are trying to hide something.
2) If the author wants me to take time, it has to be really easy to try. "Configuration" is asking me to make choices that I really have no clue about a priori. Now, maybe there is a "heavy duty" version as well. But the "straightforward" version needs to be really straightforward.
And you would be surprised how often you don't need the heavy duty version. A nice example of this in crypto is DJB's "TweetNaCl". It gets you 99% of what you need until you understand what it doesn't get you. And, it's so damn easy to include that you get very little resistance--try using tweetnacl vs libnacl vs libsodium in a project. You'll pull your hair out if there isn't a magic package already for libsodium while tweetnacl can be dumped into embedded microcontrollers without pain.
3) If the author wants longevity, simplicity is his friend. Can you even begin to run a "configure" script from even 5 years ago let alone 10? Yet, I can download an ancient Fortran 77 code file and compile it today pretty easily.
Configuration is not your friend.
Ayup. It's too annoying to translate the academic speak, but you can shortcut it.
For example, anything CS nowadays (I include crypto in this) must be 1) publicly available on the web, 2) come with running code that runs some basic tests, and 3) be limited to a single compilable file. Failure in any of the criteria means I move on. If the source code involves anything like "configure" it's an immediate fail. If the code passes all three of those criteria, it's probably more useful to read the code rather than the paper.
I used to love ISSCC for the reason that they used to demand both A) a die photograph of a chip and B) actual oscilloscope traces. You can't hide when you have to make test equipment produce data. Sadly, they got rid of that requirement in the late 90's, and the information content of the conference suffered correspondingly.