Any thoughts about higher level HDLs in embedded in software languages, like Chisel, nMigen, or others? Some other RISC-V core designers claim they've had increased productivity with those.

It seems that despite a lot of valid criticism against (System)Verilog, nothing really seems to be a on trajectory to replace it today. I'm not sure if that's purely inertia (existing tooling, workflows, methodologies), other HDLs not being attractive enough, or maybe Verilog is just good enough?

I think they're great - I earned my VLSI chops building stuff in the 90s and I can write Verilog about as fast as I can think so it's my goto language. I've also written a couple of compilers over the years so I know it really well (you can thank me for the '' in "always @()"). That's just my personal bias.

Inertia in tooling is a REALLY BIG deal - if you can't run your design through simulation, (and FPGA simulation), synthesis, layout/etc you'll never build a chip - it can take a 5-10 years for a new language feature to become ubiquitous enough so that you can depend on it en ough to use it in a design (I've been struggling with this using System Verilog interfaces this month).

If you look closely at VRoom! you'll see I'm stepping beyond some Verilog limitations by adding tiny programs that generate bespoke bits of Verilog as part of the build process - this stops me from fat fingering some bit in a giant encoder but also helps me make things that SV doesn't do so well (big 1-hot muxes, priority schedulers etc)

Possibly crazy thought. With wider CPUs needing more ports (read and write) on the register file, would it make sense to use accumulators as registers so basic boolean and math ops could be done locally with a single read port that the alu could tap?

Modern CPUs have a structure called the bypass network that lets different ALUs forward their outputs to another's inputs without having to hit the register file. It's not exactly like a local accumulator but it's something in the same direction.

Modern OoO CPUs solve the need for more physical registers than logic registers with renaming. https://docs.boom-core.org/en/latest/sections/rename-stage.h...

Yes VRoom! with renaming has as many registers as it has commitQ entries plus the architectural ones. So 64/32 + 31 (integer) + 32 (fp) + 32 (vector)

The idea was to have as many simple ALUs as registers. Results are kept in the ALU/register. All reads are essentially result forwarding. For example a simple RV32I requires 2 read ports and one write port on each register. If we use 2 R/W ports and put an ALU on each register, you reduce from 3 to 2 busses and can also do operations 2 at a time when an instruction clobbers one of its inputs. Or an ALU op along with a load/store.

err HN swallowed my * there as in: "(you can thank me for the '*' in "always @(*)")"

As an aside, the latest and active development of nMigen has been rebranded a few months ago to Amaranth and can be found here: https://github.com/amaranth-lang/amaranth . In case people googled nMigen and came to the repository that hasn't been updated in two years.

What does your benchmarking workflow look like? I am interested in

  * From a high level what does your dev iteration look like? 
  * Getting instruction traces, timing and resimulating those traces
  * Power analysis, timing analysis (do you do this as part of performance simulation) ? 
  * Do you benchmark the whole chip or specific sub units?
  * How do you choose what to focus on in terms of performance enhancements? 
  * What areas are you focusing on now? 
  * What tools would make this easier?

At the moment I'm just starting working my way up the hierarchy of benchmarks, dhrystone's been useful though it's nearing the end of its use - I build the big FPGA version (on an AWS FPGA instance) to give me a place to run bigger things exactly like this.

I currently run low level simulations in Verilator where I can easily take large internal architectural trace, and bigger stuff on AWS (where that sort of trace is much much harder)

I haven't got to the power analysis stage - that will need to wait until we decide to build a real chip - timing will depend on final tools if we get to build something real, currently it's building on Vivado for the FPGA target.

Mostly I'm doing whole chip tests - getting everything to work well together is sort of the area I'm focusing on at the moment (correctness was the previous goal - being together enough to boot linux), the past 3 months I've brought the performance up b y a factor of 4 - the trace cache might get me 2x more if I'm lucky.

I spend a lot of time looking at low level performance, at some level I want to get the IPC (instructions per clock) of the main pipe as high as I can so I stare at the spots where that doesn't happen

I'm using open source tools (thanks everyone!)

> dhrystone's been useful though it's nearing the end of its use

Would my fhourstones [1] [2] benchmark be of any use?

[1] https://tromp.github.io/c4/fhour.html

[2] https://openbenchmarking.org/test/pts/fhourstones

thanks I'll have a look - I'm not so interested in raw scores, more about relative numbers so I can judge different architectural experiments

From what little I know about microarchitecture, this seems extremely impressive. Hopefully these aren't dumb questions:

Are there GPL'd designs for PCIe, USB, etc, that could be used to incorporate this into a SoC design? If not, how much work is that compared to this?

Also, what other kind of technical considerations would be involved to make this into a "real" chip on something like 28nm?

Great questions - I'm using an open source UART from someone else, an d for the AWS FPGA system I have a 'fake' disk driver plus timers/interrupt controllers etc

So far I haven't needed USB/ether/PCIe/etc I've sort of sketched out a place for those to live - I think that for a high end system like this one you can't just plug something in - real performance needs some consideration of how:

- cache coherency works - VM and virtual memory works (essentially page tables for IO devices) - PMAP protections from I/O space (so that devices can't bypass the CPU PMAPs that are used to man age secure enclaves in machine mode)

So in general I'm after something uniquer, or at least slightly bespoke.

I also think there's a bit of a grand convergence going on in this area around serdes's which are sort of becoming a new generic interface PCIe, high speed ether, new USBs, disk drivers etc are all essentially bunches of serdes with different protocol engines behind them - a smart SoC is going to split things this way for maximum flexibility

Don't know much about the details, but this company / person claims to have developed some open source IP: http://www.enjoy-digital.fr/

Not Paul Campbell, but I'll share what I know on the matter.

So GPL'd IO blocks - This is a great question, and something I have definitely been asking myself! One thing to keep in mind is that IO interfaces like PCIe, USB, and whatnot have a Physical interface ("Phy" for short.) Those contain quite a bit of analog circuitry, which is tied to the transistor architecture that's used for the design.

That being said, A lot of interfaces that aren't DRAM protocols use what's known as a SerDes Phy (short for Serializer De-serializer Physical interface.) More or less, they have an analog front end and a digital back end, and that digital back end that connects to everything else is somewhat standardized way. So it wouldn't be unreasonable to try to build something like an open PCIe controller that only has the Transaction Layer and Data Link Layer. While there are various timing concerns/constraints when not including a Phy layer (lowest layer,) I don't think it's impossible.

The other big challenge is that anyone wanting to use an open source design will definitely want the test benches and test cases included in the repo (you can think of them like unit tests.) Unfortunately, most of the software to actually compile and run those simulations is cost prohibitive for an individual, because it's licensed software. Also, the companies that develop this software make a ton of money selling things like USB and PCIe controllers, so I'll let you draw your own conclusions about the incentives of that industry.

Even if you were able to get your hands on the software, the simulations are very computationally intensive, and contribution by individuals would be challenging ...though not impossible!

Despite those barriers, it's a direction that I desperately want to see the industry move towards, and I think it's becoming more and more inevitable as companies like Google get involved with hardware, and try to make the ecosystem more open. Chiplet architectures are also all the rage these days, so it would be less of a risk for a company to attempt to use an open source design.

I'd really be curious to hear Paul Campbell's take on this question though. He definitely knows a lot more than I do!

One advantage of SkyWater opening its PDK is Universities are starting to back fill all the hardware that is missing.

Here's a SerDes from Purdue. I don't think this particular design has been validated in silicon yet though.

https://arxiv.org/abs/2105.13256

Have you considered making an ASIC of your design? https://efabless.com/open_shuttle_program

It's likely too big for those programs - I am (just now) starting a build with the Open Lane/Sky tools not with the intent of actually taping out but more to squeeze the architectural timing (above the slow FPGA I've been using for bringing up Linux) so I can find the places where I'm being stupidly unreasonable about timing (working on my own I can't afford a Synopsys license)

Gotcha. Did you run into any issues with yosys given that it has limited system verilog support?

Ibex needed to add a pass with sv2v https://github.com/lowRISC/ibex/tree/master/syn

I'm just starting this week, I've recently switched to some use of SV interfaces and it does not like arrays of them - sv2v seems the way to go - but even without that yosys goes bang! somethings too big Vivado compiles the same stuff - I rearchitected the bit that might obviously be doing this but no luck so far.

Checkout Surelog - https://antmicro.com/blog/2020/12/ibex-support-in-verilator-... and the more recent usage in https://github.com/siliconcompiler/siliconcompiler

Do you think it is feasible to add some kind of sticky overflow detection bit for integer arithmetic, the way IEEE 754 specifies for FP? The hope is to be able to efficiently implement checked arithmetic as required by e.g. Ada. It came as a real disappointment that RiscV seems to have made that harder rather than easier, compared to the x86 and its ilk. The sticky bit is hopefully more efficient than traditional condition codes, since the compiler can emit checks for it at the end of a function or basic block, giving the hardware some time to catch up.

RISCV doesn't have condition codes, which makes building systems like this with lots of ALUs a lot easier, everything happens in the registerfile and the renaming system.

It does have 'sticky' state bits for FP and I can see how I'll implement them - the big problem is not setting them (because they can be accumulated in any order as instructions hit the commit stages), it's how you test them that effectively becomes a synchronising point in a pipe where you spend all your time trying not to do that - everything has to stop and line up in order before you can sense that state reliably.

Right, the idea of the sticky bits is that you can execute a number of instructions before hitting a sync point, as compared with traditional condition flags or exceptions which have to potentially sync on every instruction. The hope is for that have less effect on performance. I don't know if it causes issues with modern FPU's. Are imprecise exceptions (like on some old machines) still a thing? I wonder if there is a way to record the exception info so its origin can be reported even if it is not noticed til later.

If you don't synchronise it's not an issue :-) but if you have 100+ instructions in motion at any one time creating a bottleneck at the end of every basic block would likely have a big effect on how things run.

Normally we retire up to 8 instructions per clock - merging 8 bits of sticky state is some simple logic (some or gates, it doesn't matter what order they get processed in), merging 8 saved PCs (which one do you save?) is harder, it probably means a priority encoder (order probably does matter) and a 63 bit mux (still doable in a clock though)

Ah, well, you're better equipped than I am to figure this out. Anyway the FPU will have to deal with it. Maybe if the big float computations are mostly vector ops, though, that will help.

Very impressive. Do you have any experience with designing Verilog or SystemVerilog MIPS implementations as well? If so, how does that compare to RISC-V? Which one was easier in terms of design, testability, SoC integration and overall understandability?

Sorry no, I've worked on VLIW CPUs and an (unreleased) x86 core.

In general though MIPS and RISCV are similar sorts of RISC architecture, they make some of the same design trade offs (no condition codes for example), RISCV avoids some of the mistakes (delay slots for example) - I'd guess they're about the same amount of work. I can imaging making a version of my CPU by switching out the instruction decoders (probably not really that simple).

As far as SoC it probably doesn't matter - that's more of an issue of which internal buses you choose to use for memory and peripherals

SoC stuff is probably unrelated to arcitecture -

As a language VM implementor, I would really love to have a conditional call instruction, like arm32. AFAICT this would be a relatively simple instruction to implement in the CPU. Is that accurate?

yes and no - there's a few issues here:

1 - architectural - RISCV has a nice clean ISA, it's adding instructions quickly, CMOV is contentious issue there - I'm not an expert on the history so I'll let others relitigate it - it's easy to add new instructions to a RISCV machine, unlike Intel/ARM it's even encouraged - however adding a new instruction to ALL machines is more difficult and may take many years. But unlike Intel/ARM there IS a process to adopt new instructions that doesn't involve just springing them on your customers

2 - remember RISCV is a no-condition code architecture - that would make CMOV require 3 register file ports (the only such instruction that also requires an adder [for the compare]) - register file ports are extremely expensive, especially for just 1 instruction

3 - micro-architectural - on simple pipes CMOV is pretty simple (you just inhibit register write, plus do something special with register bypass) I'd have to think very hard about how to do it on something like VRoom! with out of order, speculative, register renaming - I can see a naive way to do it, but ideally there should be a way to nullify such an instruction early in the pipe which would mean some sort of renaming-on-the-fly hack

How do you feel about short forward optimizations? If I understand it correctly, BOOMv3 can convert instructions under a short forward branch shadow to be predicated.

Note I was talking about a conditionall call instruction, which is very useful for, e.g. safety checks.

conditional CALL is MUCH harder to implement well - it's because the call part essentially happens up at the PC/BTC end of the CPU while at the execution stage what you're doing is writing the saved PC to the LR/etc and the register compare (or accessing a condition code that may not have been calculated yet).

In many ways I guess it's a bit like a conditional branch that needs a write port - in RISCV, without condition codes, your conditional call relative branch distance will be smaller because the instruction encoding will need to encode 2-3 registers

I imagine something like that might be viable in the to-be-designed RISC-V J extension, as safety checks (mostly in JITs) would be close to the only thing benefiting from this.

Though, maybe instead of a conditional call, a conditional signal could do, which'd clearly give no expectation of performance if it's hit, simplifying the hardware effort required.

Yeah, I can imagine that being particularly easy to implement in VRoom! exceptions are handled synchronously at the end of the pipe (with everything before them already committed, and everything after flushed). Instructions that can convert to exceptions (like loads and store taking TLB misses) essentially hit two different functional units - a conditional exception would be tested in a branch/ALU unit and then transition into an effective a no-op or transition into an d exception and synchronise the pipe when they hit the commit stage

8080 had it too, 8086 dropped it due to disuse. In a modern context it's just a performance hack, an alternative to macro-op fusion, but for high-performance RISC-V (or i386, or amd64, or aarch64) you need macro-op fusion anyway.

Do you dance? :)

https://youtu.be/nlu0foF3WBk?t=182

I know, I'm leaning hard on that second "A" there. :D

heh! - I'm a Kiwi who lived and worked in Silicon Valley for 20 years, moved back when the kids started high school, but mostly still work there - while I was there I started a company using "Taniwha" ... great for a logo, but a mistake because of course no one in the US knows how to pronounce it (pro-tip the "wh" is most close to an english "f")

Is this a hobby? Is there a way to support the project?

What your view on that kind of design for a laptop/phone/tablet processor?

It's a personal project, but an expensive one (with very large AWS bills some months), needs a commercial sponsor to end up as a real chip you could put in real platforms and a real chip design team to build it

Maybe there's a way to hit up AMD/Xilinx for an donated or loaned FPGA box so you don't have to use AWS?

You could probably find a riscv intl member company willing to donate small fpga boards, but if this design barely fits on an AWS f1 instance, I think realistically that'd have to be a 5 digit price board

For reference, Digikey unit price for a VU440 floats around the $40-60k range

Next step up is probably a VU13P based board, there were a few on EBay a year ago ~$5k, almost bought one, but then bitcoin went up again ..... I'm hoping for a crypto crash ....

Are you making any attempt at a learning branch predictor? Is anything published about really-current methods?

Not yet - I have a pretty generic combined bimodal/global predictor - there's a lot of research on BTCs - it's easy to throw gates at this area - I can imagine chips hitting 20-30% BTC in area just to keep the rest running

My next set of work in this area will be integrating an L0 trace cache into the existing BTC - that will help me greatly up the per-clock issue rate

But can it run Quake? :D

No FP yet - that's about half (adds and multiplies) done - but no reason why not