mikeayles

The space is definitely heating up.

Flux.ai being the incumbent Silixon based out of the UK have been showing demos Quilter and DeepPCB for layout The recent OpenAI demo...

I went the other way and pre-designed sub circuits with a set of rules that allows simple merge assembly, unfortunately I didn't have the time or budget to continue with it, but I took the constrained approach as with hardware, things can't be 99%.

The one thing I will say, is that my experience of hardware engineers is very gatekeepy.

Any questions about the use of AI for any form of automation on forums/Reddit get shot down instantly and the OP's told 'skill issue' more or less. In my opinion, just as with software, most work is simple and should be automated, there will always be a need for the real engineering when it comes to high specification work, high volume optimisation and hardware debug (although Claude code is pretty handy at driving a Jlink)

I 100% agree the model is basically useless. that was never the deliverable. the artifact here is the inference engine, not the model living in it. 3.16M params at character level is just what fits in ~3MB of on-chip SRAM.

Plus I treated it as a good learning experience to get better with FPGA's but also system design.

Appreciated. and yeah, agreed the interesting comparison isn't "is this scalable as-is" (it isn't, PL fabric, cost, power), it's that the CIM / analog-compute startups you mention are chasing exactly this endpoint with real money and years of R&D, the bar for making something useful is brutally high.

However, if no-one made anything that was useless on the same thesis, a lot of these concepts would have never got off the ground. I would hazard a guess that people like taalas would have started with a (much much bigger) fpga to validate whether the approach was possible before committing to designing a chip big enough to fit an 8B model in it.

I just nerd sniped myself...

VP1902 could fit around a 500m model in, whereas a cadence protium rack of them could squeeze in a ~6B at 8bit, or a ~13B at 4bit. So accounting for the headroom of distributed compute, Llama 3.1 8B at 4bit. I don't want to even estimate how long synthesis and place and route would take on that!

Just a quick test to see if i'm back

I just posted a Show HN of my most recent side project, a live demo of a tiny-llm implemented in FPGA fabric, hitting an aggregate peak of 60,000tok/s, but a 'usable' model at 21,000tok/s

Writeup and demo here: https://www.mikeayles.com/blog/on-chip-llm-kv260/

Source and HDL here: https://github.com/MichaelAyles/kev-gpt

Show HN here: https://news.ycombinator.com/item?id=49242475

I'm using the AMD (Xilinx) K26. It's a Zynq Ultrascale+, the successor to the old classic 7000's. I'm running it on the KV260 dev board, because I'm using it for another side project.

The K26 has a quad core A53 core alongside the programmable logic (PL, or Fabric). The A53 is pretty weak, and doesn't have any hardware matmul operations, so despite the KV260 being sold as a 'vision ai starter kit' and the vitis object detection running on the arm cores, they're pretty weak cores for anything AI.

For my use, I need true determinism, so my vision pipeline is all implemented in the PL, and it was pretty disapointing that the vitis libraries are basically just opencv on linux, rather than really pushing the fabric. If I wanted probabalistic AI running on a CPU, then I sure as heck wouldn't choose a quad core A53.

Which led me to have a play with this, I saw the taalas/chatjimmy demo and wondered what I could push the fabric to.

The round trip time to DDR or CPU via AXI meant I had to keep the entire inference engine in fabric. The A53 is simply a pipe that gets a request from my server (which has a cloudflare tunnel to the real world for the live demo in the blog post) and manages a queue. So it feeds a string in, and gets a hopefully longer string back a few uS later.

It's all in verilog, because that's what i'm more used to. I did get Claude Code to do a moderate amount, as it's a side project on a side project after all, but pushing an FPGA to it's limit is definitely not as comfortable for it as it is writing a crud app in TS.

I'm using tinystories, as we are talking about megabytes of URAM/BRAM. If I used the DDR, it definitely would have been a real model, but that wasn't my goal. My goal was to hit 100,000tok/s, and even when I conceded on absolutely everything, with a token prediction size of 1tok, I topped out at 60,000tok/s. ?But increasing the window to make an actually plausible chat (story generator, it doesn't understand questions, you need to prompt it with 'once upon a time...' and it finishes it for example) I managed to break 20k tok/s.

I also created the lemmatised version, which was inspired by Kevin from the office (why use many word when few do trick) and trained a new model, I was expecting the output model to be smaller, but was suprised that it came out the same size, but it ran 30%ish faster. In hindsight it makes sense, the parameter count is fixed by the architecture, not the corpus, so training on compressed text doesn't shrink the model at all. What it does is compress the output distribution. The same story takes ~30% fewer characters to tell, so the effective speed goes up even though the per-token rate is identical. The dumbness is the optimisation.

Fully agree on publish early. The blog post with the live demo (a websocket straight to the board through a cloudflare tunnel, so you're genuinely talking to the fabric) is written and sitting in drafts while I fiddle with it. This thread is my peer pressure, it goes live in the next day or two.

It will be, I'm just not happy with my blog post before making it live. The blog will have a live interactive chat and a link to the repo with the HDL.

I don't think anything I did was particularly novel, as I really just wanted to see how fast I could push a commodity FPGA to it's limit.

Scaling to an ASIC or getting into the billions of params is where the real engineering is! This was just a side project for a side project for me while the FPGA was idle

AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.

Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.

I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two

So for people wondering if it can be used to accelerate LLM inference, sadly not.

I've been trying to hit 100,000tokens/s with a 3.28m dumb model, and even this is an order of magnitude too large to benefit.

It appears to be focussed more on latency, than throughput. Happy to be corrected?

you may find this useful: https://phaestus.app/blog/blog0031

Edit: Forgot I also got doom running in openscad: https://www.mikeayles.com/blog/openscad-doom/

and doom running in openscad in the browser at https://doom.mikeayles.com/

I benchmarked Claude Code and GitHub Copilot on the same model (Haiku 4.5) with and without RAG-powered semantic search across 60 queries on a real codebase.

RAG didn't make search more accurate on Claude Code, but it cut token consumption by 28%. On Copilot, it cut time to resolution by 44% and improved F1 by 19.5%.

The bigger finding: controlling for model, tool design alone accounts for a 30pp recall gap between the two tools. Benchmark code and data are open source.

Rewriting the backend Bitwise Cloud, my semantic search for embedded systems docs Claude Code plugin from Python to Go.

The problem was the ML dependencies. The backend uses BGE-small-en-v1.5 for embeddings and FAISS for vector search. Both are C++/Python. Using them from Go means CGO, which means a C toolchain in your build, platform-specific binaries, and the end of go get && go build.

So I wrote both from scratch in pure Go.

goformer (https://www.mikeayles.com/blog/goformer/) loads HuggingFace safetensors directly and runs BERT inference. No ONNX export step, no Python in the build pipeline. It produces embeddings that match the Python reference to cosine similarity > 0.9999. It's 10-50x slower than ONNX Runtime, but for my workload (embed one short query at search time, batch ingest at deploy time) 154ms per embedding is noise.

goformersearch (https://www.mikeayles.com/blog/goformersearch/) is the vector index. Brute-force and HNSW, same interface, swap with one line. I couldn't justify pulling in FAISS for the index sizes I'm dealing with (10k-50k vectors), and the pure Go HNSW searches in under 0.5ms at 50k vectors. Had to settle for HNSW over FAISS's IVF-PQ, but at this scale the recall tradeoff is fine.

The interesting bit was finding the crossover point where HNSW beats brute-force. At 384 dimensions it's around 2,400 vectors. Below that, just scan everything, the graph overhead isn't worth it. I wrote it up with benchmarks against FAISS for reference.

Together they're a zero-dependency semantic search stack. go get both libraries, download a model from HuggingFace, and you have embedding generation + vector search in a single static binary. No Python, no Docker, no CGO.

Is it better than ONNX/FAISS? Heck no. I just did it because I wanted to try out Go.

goformer: https://github.com/MichaelAyles/goformer

goformersearch: https://github.com/MichaelAyles/goformersearch

Fantastic comment, thanks! A review of the state of BEV's was actually going to be one of my next articles (hopefully after the additives).

Are you happy for me to drop you an email to review a draft when I'm ready?

I do now: https://www.mikeayles.com/rss.xml

Good spot, and gruez is right about the caption too (fixed both, thanks).

The car's L/hr figure was wrong. At 45 mpg (imperial) and 70 mph cruise, a car burns ~7 L/hr, not 3. That makes the flow rate ratio ~4x, which is consistent with 5x per mile and the truck travelling 20% slower.

The ~3 L/hr I originally had is what you'd see as an average over a mixed driving cycle — ~30 mph mean across urban, suburban, and motorway. I was carelessly mixing the cars combined-cycle flow rate with the truck's cruise-only figure in the same row.

The truck doesn't have this problem because a long-haul artic genuinely spends most of its operating hours in that narrow 50-60 mph cruise band. "Average fuel burn rate" and "fuel burn rate at cruise" are nearly the same number. For a car they're very different, transient acceleration, idling in traffic, and low-speed urban driving all drag the average flow rate down well below the motorway figure.

Probably the first thing to consider is the trucks have their speed calibrated periodically to ensure the accuracy of their tachographs (in the UK at least) so a truck doing 90kmph may show as 100kmph+ in a passenger car, I know my Volvo is 7% out, and my Seat is closer to 10% out.

That said, depending on the truck, there's fuses you can pull, ECU remaps and even for the older trucks with the magnetic sensor in the gearbox, the trick is/was to stick a magnet on the sensor (with a bit of string, so you can pull it off remotely if you get pulled over). All of these methods are becoming less feasible, as things like the aggregate wheel speed sensors used for ABS get used, you can't just fool one thing now.

As for the weight limit problem, that's a whole other rabbit hole!

Sorry, got mixed up there, will amend, the 60 is for +3.5t!

Edit: Nope, despite the vehicles only being able to propel themselves to 90kmph, the speed limit is indeed 60mph (in England and Wales, Scotland is a more sensible 56mph)

https://www.gov.uk/speed-limits

I have done a deep dive here: https://www.mikeayles.com/blog/on-vehicle-hydrogen-generatio...

Short answer, it takes more energy to generate than the energy it produces.

You can do things like only producing electrical power from the alternator when decelerating, ensuring no load comes off the engine, but that would require accumulation as you're not actually burning fuel then either.

But running the numbers on the power requirements, I reviewed one commercially available system (at 12v 14A) and calculated that the HHO they are able to produce is 0.037% by energy going into the engine vs regular fuel.

When presented with 0.037% of the fuel substituted, their 10-15% claim on fuel savings becomes a bit of a red flag.

Fair point that a full TCO comparison would be more complete, and it's something I'm planning to cover in a later post. But the capital cost argument actually reinforces the conclusion rather than undermining it. Hydrogen fuel cell trucks are currently 2-3x the price of diesel equivalents, roughly comparable to battery electric. The infrastructure CAPEX is dramatically higher for hydrogen (£2-5M per station vs. transformer upgrades for depot charging). And the fuel cost gap means hydrogen has higher opex too. Diesel wins today on upfront cost, yes, but hydrogen doesn't beat BEV on capital or operating costs in any current scenario I've seen. Happy to be shown numbers that say otherwise.

The big thing I haven't covered yet is HVO, which provides the WTT CO₂ saving at a slight fuel cost surcharge, which matters if a fleet is mandated to reduce their CO₂. The TCO assuming a £100k diesel truck and £200k lifetime fuel cost, a 10% HVO surcharge brings the TCO to £320k, versus a £250-350k BEV truck that costs maybe £80k in electricity over the same life. That's £320k with an 80-90% CO₂ reduction from a drop in fuel you can put in your existing trucks tomorrow, versus £280k for battery electric with zero tailpipe. Both of those are available now, with existing infrastructure. Hydrogen is asking you to spend £300k+ on the truck, £150k+ on fuel, and hope someone builds a station within range of your routes.

Good catches, both of them. The hydrogen pathway should include grid transmission before the electrolyser, you're right. In practice it doesn't change the overall ratio much (multiplying by 0.95 on both sides) but the diagram should be consistent.

The BEV cost not adjusting with the electricity slider is a bug, not a choice. I'll fix both. Thanks.

Thanks. Both good questions, and they come up a lot.

To be clear, I'm fully behind decarbonising freight. It's one of the hardest sectors to clean up and it needs serious attention. But hydrogen for road transport requires jumping in with both feet (due to infrastructure requirements) when there are dozens of smaller, commercially proven steps that get you equivalent results. Better route planning, driver training, aerodynamic retrofits, hybrid and battery electric for shorter routes, even just reducing empty running.

These aren't exciting and they don't get press releases, but they compound. The industry could cut emissions meaningfully with changes that pay for themselves today, without waiting for a national hydrogen infrastructure that doesn't exist yet.

On surplus offshore wind: the economics only work if you assume the electricity is genuinely surplus, meaning there's literally no other use for it. In practice, the UK grid still runs gas plants for roughly 40% of generation. Every MWh of offshore wind that goes into an electrolyser instead of displacing gas is a missed decarbonisation opportunity. "Surplus" renewable electricity is a future state, not a current one, and even when we get there, interconnectors, grid storage, and demand response will compete for those MWh. The electrolyser only makes sense after all of those higher value uses are saturated.

On £1.50/kg: that would genuinely change the fuel cost picture, getting you to roughly 12-15p per mile which is competitive with diesel. But the distribution problem doesn't go away at any price point. You still need compression or liquefaction, transport, and a national network of dispensing stations. The UK has 11 public hydrogen stations. Even free hydrogen doesn't help if there's nowhere to fill up. The grid is already everywhere. Adding a charger to a depot is a transformer upgrade. Adding a hydrogen station is a £2-5M civil engineering project.

The place where cheap green hydrogen gets really exciting is exactly the applications where you can't just plug in: steel, ammonia, seasonal storage, maritime. Those don't need a distributed national refuelling network, they need point to point bulk delivery to industrial sites and ports, which is a much more tractable logistics problem.

I was reading the NASA truck aerodynamics thread earlier and realised that commercial freight is one of those fields that touches everyone's daily life (everything you own arrived on a truck) but sits in a complete knowledge blindspot for most people.

I work in fleet fuel efficiency and wrote up the foundational mental model, covering why trucks weigh what they weigh, why they're all doing exactly 56mph, why diesel is so hard to replace, and why 1% fuel savings matters when you're burning 43,000 litres a year.

This is the first in a series, there's already a 2-part deep dive on hydrogen up as well. Tried to keep it accessible without dumbing it down.