Ultimately, you have to work with people you can trust. If they trust that you won't suddenly spring some bad surprise on them, and if you trust that they won't artificially pad their estimates and slack off (which is very unlikely unless you're working with some really bad apples), there shouldn't be a problem of padded estimates.