I definitely agree with and do this — it only falls apart when CI environment stuff interferes with file locations and other external state.

That said it is nice (if the scripts are simple) to do things like machine set up very easily.