well the lawsuits that Barclays and others are facing seem to indicate there was rather more harm than that. One example, from https://en.wikipedia.org/wiki/Libor_scandal

"Early estimates are that the rate manipulation scandal cost US states, counties, and local governments at least $6 billion in fraudulent interest payments, above $4 billion that state and local governments have already had to spend to unwind their positions exposed to rate manipulation."