Why not just say something along the lines of, "Sorry, I work from home and I'm on the clock but just popping out for a cup of coffee - chat later.", with a smile? It's much better for your mental constitution than holding in a scream.