It looks like availability of good quality training sets will be a stumbling block for LLM use in Verilog chip design since pretraining with other programming language corpus is not transferable. (see below for quote from Nvidia paper) A lot of high quality Verilog is locked in licensed, close source IP blocks covered by NDAs.

HDLBits problems are a toy level complexity circuits suitable for Verilog 101 course material.

I would set a benchmark for serious HDL design LLM at reaching ability to implement AXI bus components with specified by user functionality, e.g. AXI4 Slave (address, data widths, burst capability) with memory implemented as banked synchronous SRAM.

https://arxiv.org/pdf/2309.07544.pdf

* Despite the fact that multi models undergo pretraining on an extensive corpus of multi-lingual code data, they exhibit only marginal enhancements of approximately 3% when applied to Verilog coding task. This observation potentially suggests that there is limited positive knowledge transfer between software programming languages like C++ and hardware descriptive languages such as Verilog. This highlights the significance of pretraining on substantial Verilog corpora, as it can significantly enhance model performance in Verilog-related tasks *