I went through "normal" school and was always way ahead in math. My Montessori-educated sister, 3 years younger than me and not particularly mathematically inclined, could intelligently converse with me about my algebra homework when I was in sixth grade and she was in third. She wasn't as familiar with the symbols as I was, but she recognized many of the patterns (like difference of squares) in physical form.
The Montessori math approach is outlined at http://www.infomontessori.com/mathematics/introduction.htm
It seems especially dangerous to me as anti-maths types might dominate in politics and therefore might find the idea especially appealing.
Maybe it then boils down to the bigger question if talented people should be dragged down to the average levels of achievement.
Looking back, the maths taught in elementary school is very basic, too. Just multiplication and division of integers, not even fractions yet.
To be honest, I don't care that much anymore if somebody doesn't want to learn maths. It's just their loss. But maths was one of the few things I enjoyed at school, so I absolutely see no point in removing it from the curriculum.
There are some school forms (like Montessori I think, quite popular here in Germany), where pupils can apparently choose for themselves what they want to learn at any particular moment. Not sure how exactly it works, but might be a good idea.