It's trickier than just matching word sounds, the sphinx docs are first rate: http://cmusphinx.sourceforge.net/wiki/tutorialam

It is very interesting but unfortunately just appears to be too much hassle for sane people to tackle (although it'd be extremely worthwhile if someone would innovate in this space and lower the barrier to entry - most are using commercial acoustic models with the FOSS software)