The Game is the layer that very often carries the streamer, if it changes the base connection to the audience changes too. This can easy create a imbalance or conflict in interests. If the viewer can't relate to the specific game they usually leave (change of audience).

So it highly depends on your Audience. If you are a variety streamer, your audience is used to the changing environment. However this does not apply to the vast majority. If changes in audience(game) can cause up to 90+% difference its hard to quantify it as negligible factor.