My advice would be to start small. Don't quit your day job but spend some time learning more and if you feel you are ready, try getting some one off / part time gigs to apply your knowledge in a real world setting if possible. If you can afford to offer your part time efforts for free, I'd recommend volunteering to open source projects.[1] These days most popular oss projects have no trouble finding development contributions but sysadmin / infrastructure help is hard to find. Once you have done some real world work and made contacts, switching carreers should not be hard. Hang in there and keep working at what you feel you should do. I personally know a number of people who have switched carreers successfully. Don't despair.

[1] For instance: https://fedoraproject.org/wiki/Infrastructure