It does very much matter which direction the black hole is moving; it needs to be moving toward you or you won't get any blueshifting effect, which is the entire reason the energy budget works out.