It depends if you are ok with 1/3 to 2/3rds of your visitors bouncing due to loading times and losing 3 to 5x of convertion rate depending on sources...

No need to be sarcastic. What page load speed at "slow 3G" speeds do you believe is necessary to avoid that?

(I work for a non-profit cultural heritage organization, we don't really have "conversions")

I didn't mean this sarcastically, it is a decision and may not apply to all situations.

You can see these kinds of differences with just a few seconds difference, ideally I aim to stay under 2s, even on the slowest connection type. 2s is already very long for a user to wait and many will not.

Non profits are tricky. You could see volunteer sign ups and donations as conversions. I manage a non profit site as well and unfortunately I don't have a good solution that is both fast and approachable for our staff to use, so we had to make that compromise as well.