I thought this was a rumour started by Feynman himself in order that the authorities would think twice about asking him to preside over something or other.

The 'his calculus or his drum playing' part is kind of a giveaway.