It's associated symbol is literally a pointer directly to a memory location
A cons is not "raw memory"; it is a typed data structure, and in Common Lisp it is even considered an OOP object of the cons class. It is equivalent to having a data structure called "pair" that can hold two things.
All memory-allocated data structures ultimately have an address of some kind, so this whole discussion about pointers is utterly meaningless. For that matter, arrays, strings, and basically every known memory data structure is "raw" too. After all, they have an address.
The fact that cons/car/cdr can be implemented cheaply in hardware stems from the fixed and small size of cons. That doesn't make it somehow inferior.
By the way, the Java stuff is implemented in hardware too, does that make Java inferior? http://en.wikipedia.org/wiki/Java_processor
Why are there no Fortran machines or Algol Machines or C machines?
The `cons` cell is a raw memory structure, just one step removed from assembly and originally accessed directly in machine code with `car` and `cdr`. It's associated symbol is literally a pointer directly to a memory location. It's so close to assembly language that thirty years ago implementing it directly in hardware was considered practical.