You're correct that neural networks don't encode information in the way current digital machines do (i.e. with explicit records/files/etc encoded in bytes). I would disagree that this is the only useful view of information. And certainly, from a Shannon information perspective, the network structure contains information (in the Shannon sense) about the environment or problem for which it was optimized.
... the way a map represents a territory instead of how a guide book does ;)
Or the way a 3D structure made out of the same 20 joined amino-acids could represent either an enzyme (the code) or a protein (the data) and that there is no distinction between code and data (which is the great insight realized in the original Lisp. It is not a coincidence that Lisp was the language or the classic AI. Sorry for the reminiscence - just love this part).
I think use of the term 'information' is not accurate. There are no bits and the whole information theory, it seems, is synthetic and has nothing to do with how the brain actually works. It is not a digital machine but electro-mechanical, if you wish.