This is a genuinely rare experience. Was it like old hardware on a shelf that no one knew what to do with and you tried it out? Or this was how lessons/clubs/assignments got things done, using an antique IBM tabulator?
When I started public high school near Boston in the early 80s we didn't have Apples or TRS 80s or PCs, it was terminals that used printed output ... can't remember the details, maybe DECwriter to PDP 11.
strikes me as gilding the lily (I earlier even imagined possibly even a claude-esque stock 'one caveat' flourish, but I know you wouldn't do that given your background). Addressable memory, where any given addressed memory register can contain not just conventional data to be operated on but can also contain an address of some other memory register to jump to. is what strikes me as the highly-enriched uranium of computer architecture. indexing an array is a feature bump by comparison.
In the early computers with non-pipelined execution and without cache memories, and also without security concerns, modifying the executable program during execution did not have disadvantages.
Nonetheless, pipelined execution became widely used and also cache memories appeared soon, and such performance-enhancing techniques could not work well with self-modifying programs, so index registers became absolutely necessary in fast CPUs.
Previous pre-computer programmable machines, back to the Jacquard loom, stored programs in a completely different medium than the data. Plugboards, chains of Jacquard cards, cams, etc. Von Neumann insisted that programs were just data and belonged in the same memory as the data. This was a conceptual advance. The hardware had to get there to make it work.
The next big innovation was the stack, in the modern sense. This came later than one might expect. Note that there are at least two types of stack computers. There are ones with a return point and context stack, which is the C view of the world. And there are ones where working storage is on the stack, like Forth machines and early Burroughs machines. The first kind seems to have been invented for ALGOL in 1957, because ALGOL allowed recursion. Stacks made a lot of people nervous back then, because how much memory to allocate for the stack was unclear, and there wasn't much memory. FORTRAN disallowed recursion and thus did not need stacks and stack frames.
So storing into the program hung on in some architectures. UNIVAC 1107 and later mainframes had a Store Location Jump (SLJ) instruction. This stored the return address into the beginning of a subroutine. Not into a stack frame; there was no stack by default. Into the first word of the subroutine's code. For non-recursive non-concurrent code, it worked.[1] But it was used less over time, with different constructs used instead.
Program in the same memory as data, yes. Program can create more program, yes. Program modifies itself while running, not so good. Index registers eliminate most needs for the third case.
[1] https://www.fourmilab.ch/documents/univac/instructions.html
An iconoscope was a cathode-ray tube used in video cameras, where it allowed the reading of every pixel of an image, and where the value of the pixel consisted of the electric charge stored at that point (the amount of charge depended on the light exposure of a photoconductive plate). Williams did exactly what von Neumann had suggested, combining the reading of the memory like in an iconoscope tube with the writing of the memory like in a CRT display tube.
The first commercial IBM fully-electronic computers, IBM 701 for scientific computing and IBM 702 for business data processing, used Williams tubes, according to the von Neumann proposal. The Williams tube memory of the IBM computers was an advantage over the Univac computers, which were still using the slow delay line memories invented by Eckert and Mauchly (which had been derived from the delay lines used as analog memories in radars, during the war).
The next generation of IBM computers switched to the magnetic core memory invented at MIT, except for the cheapest computers, like IBM 650, which used magnetic drums for the main memory. IBM also invented soon the magnetic disks, as a memory tier intermediate between magnetic drums and magnetic tapes. The use of magnetic tapes had started in the first commercial electronic computer from USA, UNIVAC I.
By the end of the fifties, some computers had 5 tiers of memory, from the fastest to the one with the greatest capacity: registers made with flip-flops, magnetic cores, magnetic drums, magnetic disks and magnetic tapes.
Eckert and Mauchley came from a background of trying to get compute done with modified IBM hardware. They were painfully familiar with the lack-of-memory problem. Everybody involved with that era of machinery was. (I've used IBM plugboard wired tabulators, like the one at the Computer Museum in Mountain View. My high school had one.) The ENIAC had a workaround for lack of memory, huge rolling cabinets of manually set rotary switches. If something better had been available, it would have been used. You have to appreciate how much early architecture was restricted by available components.
The first real hardware breakthrough was acoustic delay-line memory. First mercury tanks, then long coils of steel wire. Once that was available, it was clear that was the place to put the program. But it took a while to make that work reliably. Then there was Williams tube electrostatic storage, and high speed drums, which got things going. Sort of. Memory cost was an issue well into the 1980s. In the early 1970s, a megabyte cost a million dollars. Cheap memory didn't appear until the 1990s.
The big lack in Von Neumann's architecture was index registers. He was into storing into the address part of instructions. The Manchester Mark I had the first index register (the "B box"), at which point programs could be read-only and still be able to index arrays. That completed the basic CPU architecture.