I don't know what you're trying to say here. NNs can have plenty of context. Let's take just the Transformer architecture for concreteness, ignoring all the others. Something like GPT-2 uses a fixed window, but it can be expanded arbitrarily widely if you're willing to pay the compute price; the scaling can be improved within that window by not computing attention over the full window, per Sparse Transformers; recurrency can be added to make it more RNN-like, as does TransformerXL; compressed versions of the memory can be stored and fed back in, allowing lookback arbitrarily far, per Compressive Transformers; multi-modal input, like images + embeddings + text, are of course completely possible and can represent arbitrary metadata and context; there are many different ways to add in external memory stores (DeepMind regularly experiments with these, like the DNC); and so on. What exactly do you think is impossible in principle in the connectionist paradigm to make NNs do (and why do you think your brain is exempt from this)?
Maybe some parts of image recognition require spinal cord neurons and muscle tissue to interact? So is it just count of inputs to neuron, or there are ways where inputs to neuron are modified by other chemicals in a body? Are we making weights on inputs stronger because of good reasons, are making some other inputs weaker because of good reasons?
From what I see and understand, even if you have multiple layers of perceptrons and back propagation. It is still really "deeply lacking" because you are not able to train networks with all context there is. You can still overtrain network, so more training is making it actually worse. You can train NN only so much, just like you cannot teach dog to speak, it can perfectly fine understand your commands but you will never communicate with the dog on a level of "this painting is beautiful, you agree?".
I also do not believe in possibility of creating AGI.