Having a rough understanding of what memory is at a very rough high-level grasp ("numbered byte-size cells" isn't very in-depth after all) doesn't preclude one from working "closer to one's problem domain", likewise lacking such grasp doesn't bring one any closer to one's problem domain.. what am I missing?
In the book, "Mythical Man Month", Fred Brooks talks about the two kinds of complexity in solving problems with computers. There's the "essential complexity" that's there because the problem you are solving is actually complex, and then there's the "accidental complexity" that's not actually required to solve the problem, but to make the computer happy.
Let's compare C arrays and Python/C# Lists. For my pretend problem, I need an ordered, index-able set of things.
With the C array, I need to know how big to make it before I create it. I need to allocate the memory that is used before I use it, and I must deallocate that memory when I'm done with it. I might underflow or overflow the array and allow Eastern European hackers control over my server and data. Even if I checking for overflows and underflows in all the proper places, I also have to add code to every one of those places handle each possible error.
Whereas in C# and Python, I just make a new list and stick things into it. When it goes out of scope, it disappears automatically.
Python and C# move me closer to the actual problem being solved by removing this "Accidental Complexity". When complexity removing is done well, it also removes the requirement for the programmer to know about numbered cells of memory, because numbered cells of memory is "Accidental Complexity" for the problem domain.
Now when I know how the computer is actually working, it does bring benefits. Today, in fact, I'm writing C code for an embedded ARM chip. Knowledge of memory numbers is a little required here. But for most software developers, knowing how memory works is not a requirement for working software. Even memory-as-a-numbered-set-of-cells is still just a huge abstraction from how the memory is actually being handled by the hardware.
But that is essential complexity when you need to reason carefully about memory use.
If you need to understand memory to use a language, then it's not abstracting well enough.
It's a good thing. Most of the time for most of the people, it's better to work closer to your problem domain than to the CPU.