I will say that the Swedish population has become more polarised over this question - both the number of people who claim "immigration is good for Sweden" and "immigration is bad for Sweden" has risen over the past 10 years, shrinking the middle. There is also a strong correlation between education and attitude to immigration (in 2011, 56% of the people who had only 9 years of education agreed that reducing the number of asylum seekers to Sweden is good while only 22% of those with a university-degree agreed with the statement). This might explain some of the frustrations showcased by the people supporting Sverigedemokraterna and other anti-immigration parties - they might simply lack the ability to debate in the way public debates are typically held and coherently express their ideas in a convincing way.
The lack of immigration debate in the 1990s is what caused Dansk Folkeparti to gain enough support, that from 2001 to 2011, they could control the Danish government. So even if Sweden's immigration debate is happening, immigration debates always happen too late.
(But this is not unique to immigration.)
Sverigedemokraterna is big because a large slew of the Swedish population is slanting toward racism and are prone to accept simple "solutions".
There are also other factors like that some people genuinely seem to believe that SD has cleaned up their act.
EDIT: I am personally strongly pro-immigration (much more so than any party in Sweden) and I have to disagree about there being much debate about immigration in traditional media. There is a lot in social media though, so I have no idea where all the racists get their idea that they cannot debate immigration issues anywhere.
Because in Denmark, immigration can be part of public debate without being shunned, while in Sweden, it's hard to get any sort of ground in this debate without being labelled a racist.
So now, some of them are trying to be more 'reasonable' about it with the party Sverigedemokraterna. I assume because they haven't made any progress publicly in Sweden by just being seen as extremists. And more and more figures are beginning to tell Swedes that immigration is not an issue you can ignore, thus Sverigedemokraterna's support is rising, because it's the only party that wants to talk about it.
But I am confident, that most of its votes are 'protest votes', to get the other parties to talk about it.