That's just consonant mutation[0][1], like how english speakers say "a pencil" but "an umbrella". (Ie, "hon", "pon", and "bon" are all the same word, just pronounced differently due to environment.) The fact that ほ ぼ ぽ (ho bo po) are all the same underlying letter, just with different diacritics, kind of hints at this.

0: https://en.wikipedia.org/wiki/Consonant_mutation

1: https://en.wikipedia.org/wiki/Rendaku