No, the point is that your code is the derivative work. If you distribute it to the customer with the intent of it being a useful integration, you can only do so legally under the GPL. The GPL library owner could sue you to stop that distribution.

Obviously there's some gray area if you, say, publish your code in the context of fair use (as an academic paper, say) and someone else does the integration work without your blessing (this is what I meant by "useless piece of source code"). But simply saying "build it yourself" is not a get out of jail free card for the GPL.