I've just always be impressed by the amount of energy that boats impart when they collide with anything. Like in this case, the entire bridge was brought down by a collision at under 8knots. Obviously that ship had orders of magnitude greater mass than the little pilot boat, but presumably you want to avoid even dinging up the larger ship, and so that's an orders of magnitude smaller goal as well.
Back to the pilot boat: Steering one in heavy seas when trying to deliver or recover a pilot is clearly a difficult art. But at least in the videos shown, there doesn’t seem to be much of an impact between the vessels. I am sure that is in part due to the skill of the steersman. Besides, I am sure the pilot boat will have permanently mounted fenders all around.