Depending on what you want to program, lack of math knowledge might mean you'll have to dig a little deeper into some subjects at first and gain that missing knowledge. But that's it. Just some extra time. Go for it :)