Architectures are also tweaked slightly although they continue to be largely based on the two architectures that started the current deep neural net boom, Long-Short Term Memory Networks (LSTMs) and Convolutional Neural Networks (CNNs), which are largely used for natural language processing and machine vision respectively.

Every once in a while there is a bit of excitement about results that come out of adding a new parameter which is then given a fancy name like "attention" and generates much breathless copy about how neural nets will soon become as intelligent as a dog/ your three year old/ your personal assitant etc. Usually though it's not clear whether these new parameters or architectural tweaks are really responsible for the success of the proposed techniques and when one looks at the field as a whole it becomes more and more apparent that the most successful work is backed by the most data and the most computing power.

To slightly er tweak your prediction, I think what'll happen is that, indeed, like you say, there will be libraries and APIs and so on (well, there already are) but all these will be controlled by large companies that can afford the ground work. So, I don't expect small outfits, research teams at universities or tiny startups, to make a big difference. Academia is fast losing the ability to produce work that beats the state-of-the-art anyway (that's from personal comms with other researches). So in terms of academia perhaps we should expect a shift away from neural nets, to something that's easier to do research on.