We have had visa free travel within East Africa for a long while now, and nothing massively horrible has happened, despite the other countries in East Africa having it, to various degrees, worse compared to Kenya. Plus Kenya already spent decades hosting millions of refugees fleeing wars from South Sudan and Somalia, again to no massively horrible effect.