This kind of low-level work is significantly more complicated than even most kernel developers realize - hence the need for articles like OP. Ditto for anything on large (more than single-board) systems. The intersection of the two was, frankly, a bit exhausting. Just keeping track of all the moving parts and their respective states induced a cognitive load that made debugging other already-hard problems that much more difficult. My hat's off for anyone who has kept on doing that stuff longer than I did, or who has to do it in an environment where vendors are keeping so many secrets.
TL;DR, the last line is "Once all of the above is completed, the processor will be able to successfully fetch instructions from a boot source. You are now effectively at the same point you would have been 5 months ago, had this been a standard 750 bringup... Board bringup from this point should be very straightforward and follow established methods."
My university (University of Colorado Boulder) bought one of a very few SiCortex systems ever sold. As an undergraduate I competed at SC07 and SC08 in the Cluster Challenge competition.
Our coach was a CU facility member, Doug, who also was responsible for the SiCortex box we bought. At SC08 he told us that another one of the teams was competing with a SiCortex box. We knew that they would win the LINPACK part of the challenge, but we didn't know that LINPACK was basically the only thing they managed to get working.
We had also heard rumors that SiCortex was in trouble financially at that time. When we were walking the show floor at SC08, we came across the huge SiCortex booth, which had 10 or so machines of different sizes (I believe the smallest was a 64-core workstation and the largest was a ~5000 whole rack system).
I remarked to Doug that SiCortex didn't look to be in such bad shape.
Doug turned to me and said, "25% of the machines SiCortex has ever made are in that booth".
The SiCortex idea was like VLIW. On paper the numbers look great. On highly optimized synthetic benchmarks it looks good. On real world code you find out how hard it is to get good performance.
Also the machine positively sucked for linear integer code - like, say, compilers or OS kernels. One of the first things customers would do, naturally, was compile. Bad first impression. Also, it was nearly impossible to get a Lustre MDS for a thousand-node cluster (which is what the biggest machine was) to run for any length of time without falling over, because Lustre was designed around the assumption that the MDS would be bigger and beefier than anything else and have "poor man's flow control" in the form of a relatively slow network. In our case it was exactly the same and completely unprotected because the interconnect was the fastest part of the system by quite a margin. That was my nightmare for those two years. I've heard that Lustre has since added some flow control ("network request scheduler") but I was never able to benefit from that. PVFS2 worked better, and Gluster (which I worked on for nearly a decade afterward) would probably have been better still because it's more fully distributed and less CPU-hungry.
The reason I mention all this is that there's an important lesson: building a system with a very unusual set of performance characteristics is a terrible idea business-wise, because people won't be able to realize its potential. Not even in a fairly specialized market. They'll just think it's slow. Unless it's truly bespoke, literally a one-off or close to it, nobody will want it.
P.S. I actually had to visit CU-Boulder to debug something on that machine, with the aforementioned Doug. It became one of my favorite "war stories" from a 30-year career, but this has gone on long enough so I'll skip it.
(And that being said, now I’m wondering whether you could force eviction and retainment into L3 cache on demand, to achieve something like memory bank switching…)
It's not really clear to me from the limited bits of info that I've read whether or not L3 is guaranteed to be accessible when doing CAR, but, if it is, you've got enough memory available to do a lot of stuff. (And even the L2 cache is starting to get pretty big on the higher-end current-gen chips.)
(This is why I compared to the SNES: if you have to map the SNES's RAM and [every bank of] the game's ROM, then you're looking at 4–16MB depending on the game. The SNES is pretty much the newest console whose games would entirely fit, I think.)
NB L3 is unified most of the time, but but with L2 you still need to distinguish between data/code.
But I found out what Itanium 2 did had a split L2 cache.
The Zen 4-based Genoa-X will have up to 1 GB of L3 cache! You could comfortably fit Windows 95 there...
So I can say with some authority that it is possible to have a complete running installation of Windows 95 in 16 megs of disk space; however I had to trim it pretty brutally to fit, so, for example, I think it had 2 fonts left, one of which was used for the widgets that display window-close boxes and so on.
The problem was that the object of the exercise was the benchmark how much quicker it was — but there wasn't enough space left to install any applications, and our benchmark tool used real applications. So, although I did several days work and I did get paid for it, the effort was at heart in vain. But in principle yes you could run Windows 95 entirely out of 16 MB of cache memory, and if you had say 32 MB of cache memory it would be no problem at all.
That's with them holding back and not adding any v-cache. If you stacked an extra 64MB on each of the 12 compute dies, you'd have 1152MB.
The somewhat surprising but true implication is that on boot, the CPU is initialized before the RAM is initialized. So there is a window of time during boot when the main core on the CPU is running instructions that cannot access the RAM. Even on register-starved x86 it is possible to write code without using RAM, but it certainly seems more convenient to me to treat the cache as RAM.
Documentation for a special compiler that compiles to code that doesn't use RAM: https://github.com/wt/coreboot/blob/master/util/romcc/romcc....