Maybe a dumb question, but are you on windows? Check that Defender isn't scanning every single time data is written to your veracrypt drive. When I was on windows, the most painful slowdowns were almost invariably caused by Windows Defender.