>The impression I get is that you must be someone incredibly annoying to work with

I don't get this impression at all, but in any case it seems an unnecessary and rude remark.