HDLBits problems are a toy level complexity circuits suitable for Verilog 101 course material.

I would set a benchmark for serious HDL design LLM at reaching ability to implement AXI bus components with specified by user functionality, e.g. AXI4 Slave (address, data widths, burst capability) with memory implemented as banked synchronous SRAM.

https://arxiv.org/pdf/2309.07544.pdf

* Despite the fact that multi models undergo pretraining on an extensive corpus of multi-lingual code data, they exhibit only marginal enhancements of approximately 3% when applied to Verilog coding task. This observation potentially suggests that there is limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog. This highlights the significance of pretraining on substantial Verilog corpora, as it can significantly enhance model performance in Verilog-related tasks *

Maybe someday the LLM's will just read the spec and come up with new designs for which they have no examples!

But I agree with pbazarnik that their benchmark are still tackling with toy-level problems. Bits operations, Karnaugh map and other stuff are just one-level of transformation logic, really want to see more in-depth examples such as high-speed circuit optimization (say a clk over 600MHz), complex filters and PID controls, etc. Wouldn't hurt even if they only achieve 20% correctness on these problems, as long as the approach can work once every few times.