Somehow the mainstream parties and especially the left have shifted from anti-racism to a blanket and absolute "immigration is good" and "all refugees welcome" consensus (and not only that but in some cases a stance that more diveristy should actually be encouraged) and anyone raising concerns is indeed labelled as "right-wing" or "far-right", which is not helpful at all.
And by and large it's not like centre- or even left-wing parties are in favour of "all refugees welcome" or "[all] immigration is good". Actually Netherlands has had a broadly right-wing government for about 20 years now, which has generally been against immigration.
And if we look at France, Germany, UK, then the situation is roughly similar (with differences in details and nuance, of course).
If I take the example of the UK: arguably right-wing government for 14 years with grass root Conservative backers against immigration. Well, last year the UK had the highest legal immigration on record.
Considering the popularity of Wilders in the Netherlands perhaps the government did not actually do anything to limit non-EU immigration, either...
If the people feel taken for a ride this cannot end well.
And I do think "how much migration is too much migration?" is a serious question worth asking; there's a big difference if 1%, 10%, and 50% of the population are migrants. It's a shame this topic has been hijacked by right-wing firebrands.
One of the main problems is just housing; you can have long discussions on the cause(s) of that and I don't think migration is the root cause, but at the same time it's hard to deny that migration puts extra pressure on things, especially in the Netherlands with the whole tax discount for migrants. And abstract discussions are all fine, but if people ain't got no place to live in, then well... Long-term plans and discussions don't really resonate if the short term sucks so much.