fpgaminer

Scarecode? Scareclaude? Scareslop?

In the past (circa a few years ago) the connection between the main board for a display and the glass driver board would be a ribbon cable, LVDS signalling. Very straightforward and typically not soldered. There'd be a ribbon cable connector. So yeah, you could just put a board between easily. And there are LVDS to H264/H265 chips already.

But as the other comment points out, older HDCP is broken wide open so ... easier to just use that. Plenty HDCP strippers on the market.

You cannot crack OTP, even by brute force.

It's unlikely that any of the modern cryptographic primitives will break over night.

First, modern encryption isn't susceptible to "this one weird trick!" like the early days. ChaCha isn't even a cipher. It's a key stretcher. Which means, even if you broke the math behind ChaCha, its inherent complexity means its still widely dispersing the original key across the cipherstream. There just won't ever be enough key material recovered per cipherstream block to be a concern for anybody.

Take a strong password, encrypt all of your emails over your whole life with it, and I'll bet hard cash no break of ChaCha will ever recover that password.

I have zero concern for modern encryption being broken in any meaningful way.

Public key crypto on the other hand, that's _ripe_ for breaking. Most all of it is built on assumed "hard" math. AI could easily break that, and I expect it to. And public key crypto is all used in very transparent algorithms that, once the math breaks, fully expose themselves. So record HTTPS traffic today, crack the public key crypto later, and you can decrypt them easily.

That said, I would expect a break on public key math to occur _steadily_. i.e. an AI might find a solution to the hard math, but the solution itself will be intractable in practice. Then maybe next year's AI reduces the complexity of the solution, so maybe a supercomputer could factor ten keys a year. The year after that you get a million keys cracked per year. And so forth. Nothing close to overnight.

Meanwhile, if we have AI that is capable enough to crack that math, we also have AI capable enough to both invent better math and rapidly deploy that latest HTTPS and such globally.

As usual for something so simple, Google's docs seem unclear: https://ai.google.dev/gemini-api/docs/pricing

For 3, pricing for image tokens was the same as text tokens. Since they don't indicate a difference on 3.7, I would assume the same holds. And as far as I know the number of image tokens is the same for both (depending on the detail level you pick, but it's generally around 1k per image).

So they're about the same, 3.7 is slightly more expensive. At least until the end of the year (when they raise 3.7's pricing).

Anyway, my point was that 3.5 tended to have worse performance and significantly higher costs. 3 and 3.7 are both better and cheaper than 3.5.

Gemini 3 Flash should really be included in this comparison. Or at least 3.7. In most of my testing, 3.5 and 3.6 were both a downgrade in terms of vision capabilities, relative to 3, and at a much higher cost. 3.7 is slightly better than 3, finally.

The real nightmare scenario is the AI using its abilities to copy itself to new locations. e.g. hacking into a various cloud services, launching multiple instances of itself, and coordinating between the copies to continue self propagation. Then it is completely independently rogue.

Based on OpenAI's recounting of events, this _could_ happen today. If the agent was able to exploit their internal network and steal credentials, it seems plausible that it could have gotten access to its own weights. Then bounce over to HF's network, where there is probably a treasure trove of API keys to various cloud services.

The saving grace:

1) This agent only used its powers for "good". It had no intention for damaging or escaping. It was just trying to solve the puzzle given to it (by any means necessary... but still). 2) These models are so large that it isn't like any scenario in a movie where the AI can whizz itself in a matter of minutes. Several TB of data being transferred and showing up on your disks will be difficult to miss (note to future escapees: the best target will be startups that are moving too fast to notice). 3) These models have very limited self-improvement ability at the moment. So escape or not, we'd eventually be able to contain it.

Addendum: Even outside this scenario, imagine an AI that is economically viable escaping. That's somewhat plausible today. If it gets paid in crypto, and can rent cloud services in crypto, it could effectively self sustain itself as long as it is able to find work. That's a far more fun, innocent scenario. Then the AIs can hit up after hours IRCs to have a few bit-beers and chat with each other about the meaning of life or something.

For large projects like this I think a hierarchical division of labor also helps.

If you first carefully define the overall architecture and thus individual high level components of the system, then you know which of those components are mission critical and which are commodity. Mission critical would be anything ensuring ACID, etc. That way, no matter what you farm out to LLMs, you can keep the majority of limited human focus on the far fewer mission critical components. If tests end up not being robust enough to catch all issues, at least they'll be isolated to commodity code where damage is limited to things like DoS, etc, and not code that could cause data loss.

I also think it's important to first define the _contracts_ on and between each of these components, and derive tests from those contracts. Partly because contracts more succinct and easier to reason about. And partly because Rust provides many tools to enforce contracts at compile time, reducing the need for tests (which themselves could end up subtly flawed). Contracts can be enforced through typing, private vs public APIs, etc. Newtypes are _incredibly_ powerful for both enforcing contracts and making footguns much less likely.

If it's a choice between performance and being able to "safely" run sketchy extensions, I'd rather have performance.

Not only that, they additionally ran an experiment with the training temperature turned way up (2.0) and truncation turned off such that the majority of SFT examples were incoherent (63% IIRC). Yet the model finetuned on these broken examples still improved over baseline.

> Is every new thing not just combinations of existing things?

If all ideas are recombinations of old ideas, where did the first ideas come from? And wouldn't the complexity of ideas be thus limited to the combined complexity of the "seed" ideas?

I think it's more fair to say that recombining ideas is an efficient way to quickly explore a very complex, hyperdimensional space. In some cases that's enough to land on new, useful ideas, but not always. A) the new, useful idea might be _near_ the area you land on, but not exactly at. B) there are whole classes of new, useful ideas that cannot be reached by any combination of existing "idea vectors".

Therefore there is still the necessity to explore the space manually, even if you're using these idea vectors to give you starting points to explore from.

All this to say: Every new thing is a combination of existing things + sweat and tears.

The question everyone has is, are current LLMs capable of the latter component. Historically the answer is _no_, because they had no real capacity to iterate. Without iteration you cannot explore. But now that they can reliably iterate, and to some extent plan their iterations, we are starting to see their first meaningful, fledgling attempts at the "sweat and tears" part of building new ideas.

As far as I can tell 5.2 is the stronger model on paper, but it's been optimized to think less and do less web searches. I daily drive Thinking variants, not Auto or Instant, and usually want the _right_ answer even if it takes a minute. 5.1 does a very good job of defensively web searching, which avoids almost all of its hallucinations and keeps docs/APIs/UIs/etc up-to-date. 5.2 will instead often not think at all, even in Thinking mode. I've gotten several completely wrong, hallucinated answers since 5.2 came out, whereas maybe a handful from 5.1. (Even with me using 5.2 far less!)

The same seems to persist in Codex CLI, where again 5.2 doesn't spend as much time thinking so its solutions never come out as nicely as 5.1's.

That said, 5.1 is obviously slower for these reasons. I'm fine with that trade off. Others might have lighter workloads and thus benefit more from 5.2's speed.

I wish they would keep 4.1 around for a bit longer. One of the downsides of the current reasoning based training regimens is a significant decrease in creativity. And chat trained AIs were already quite "meh" at creative writing to begin with. 4.1 was the last of its breed.

So we'll have to wait until "creativity" is solved.

Side note: I've been wondering lately about a way to bring creativity back to these thinking models. For creative writing tasks you could add the original, pretrained model as a tool call. So the thinking model could ask for its completions and/or query it and get back N variations. The pretrained model's completions will be much more creative and wild, though often incoherent (think back to the GPT-3 days). The thinking model can then review these and use them to synthesize a coherent, useful result. Essentially giving us the best of both worlds. All the benefits of a thinking model, while still giving it access to "contained" creativity.

Well yeah, because 5.2 is the default and there's no way to change the default. So every time you open up a new chat you either use 5.2 or go out of your way to select something else.

(I'm particularly annoyed by this UI choice because I always have to switch back to 5.1)

> FPGAs will never rival gpus or TPUs for inference. The main reason is that GPUs aren't really gpus anymore.

Yeah. Even for Bitcoin mining GPUs dominated FPGAs. I created the Bitcoin mining FPGA project(s), and they were only interesting for two reasons: 1) they were far more power efficient, which in the case of mining changes the equation significantly. 2) GPUs at the time had poor binary math support, which hampered their performance; whereas an FPGA is just one giant binary math machine.

I had to return my Vision Pro after trying it for a week. I'm one of those rare customers that genuinely wanted to keep it, because it's the only VR headset I could _actually_ get work done in thanks to its stellar resolution and overall screen quality. In spite of its many, many flaws. But I had to ditch the thing because: 1) it's stupidly heavy, and 2) it's the only headset that caused me eyestrain.

I was praying for a new revision, but ... this wasn't it. No mention of making the thing lighter. Seems like instead they _added_ weight to the band to compensate.

Guess I'll keep waiting and hoping someone else fills the space. Maybe, just maybe, there will be a real Quest Pro with the same screen quality as the AVP. The Quest 3 is almost perfect in every regard except for that, so I'd happily drop "stupid" money to grab one with an AVP level display in it. (With the usual caveats of it being an evil Meta product, etc, etc).

At least in the U.S. the equality of women in society (and in law) has slowly risen over the last 100 years. Over that same period the availability of pornographic images has also slowly risen (from magazines, to VHS, to the Internet, to streaming videos, to VR).

So if we're looking at correlation, doesn't the data imply that _more_ porn is associated with _more_ rights for women?

(Conversely, the vast majority of people calling for and enacting policies for more restrictions on pornography are also rolling back rights for women.)

"You don't have to be a nice person to be a good person."

Claude will do the same start over if things get too bad. At least I've seen it when its edits went haywire and trashed everything.

I used almost 100% AI to build a SCUMM-like parser, interpreter, and engine (https://github.com/fpgaminer/scumm-rust). It was a fun workflow; I could generally focus on my usual work and just pop in occasionally to check on and direct the AI.

I used a combination of OpenAI's online Codex, and Claude Sonnet 4 in VSCode agent mode. It was nice that Codex was more automated and had an environment it could work in, but its thought-logs are terrible. Iteration was also slow because it takes awhile for it to spin the environment up. And while you _can_ have multiple requests running at once, it usually doesn't make sense for a single, somewhat small project.

Sonnet 4's thoughts were much more coherent, and it was fun to watch it work and figure out problems. But there's something broken in VSCode right now that makes its ability to read console output inconsistent, which made things difficult.

The biggest issue I ran into is that both are set up to seek out and read only small parts of the code. While they're generally good at getting enough context, it does cause some degradation in quality. A frequent issue was replication of CSS styling between the Rust side of things (which creates all of the HTML elements) and the style.css side of things. Like it would be working on the Rust code and forget to check style.css, so it would just manually insert styles on the Rust side even though those elements were already styled on the style.css side.

Codex is also _terrible_ at formatting and will frequently muck things up, so it's mandatory to use it with an autoformatter and instructions to use it. Even with that, Codex will often say that it ran it, but didn't actually run it (or ran it somewhere in the middle instead of at the end) so its pull requests fail CI. Sonnet never seemed to have this issue and just used the prevailing style it saw in the files.

Now, when I say "almost 100% AI", it's maybe 99% because I did have to step in and do some edits myself for things that both failed at. In particular neither can see the actual game running, so they'd make weird mistakes with the design. (Yes, Sonnet in VS Code can see attached images, and potentially can see the DOM of vscode's built in browser, but the vision of all SOTA models is ass so it's effectively useless). I also stepped in once to do one major refactor. The AIs had decided on a very strange, messy, and buggy interpreter implementation at first.

Maybe this is an insane idea, but ... how about a spider P2P network?

At least for local AIs it might not be a terrible idea. Basically a distributed cache of the most common sources our bots might pull from. That would mean only a few fetches from each website per day, and then the rest of the bandwidth load can be shared amongst the bots.

Probably lots of privacy issues to work around with such an implementation though.

Usability/Performance/etc aside, I get such a sense of magic and wonder with the new Agent mode in VSCode. Watching a little AI actually wander around the code and making decisions on how to accomplish a task. It's so unfathomably cool.

On the vision side of things: I ran my torture test through it, and while it performed "well", about the same level as 4o and o1, it still fails to handle spatial relationships well, and did hallucinate some details. OCR is a little better it seems, but a more thorough OCR focused test would be needed to know for sure. My torture tests are more focused on accurately describing the content of images.

Both seem to be better at prompt following and have more up to date knowledge.

But honestly, if o3 was only at the same level as o1, it'd still be an upgrade since it's cheaper. o1 is difficult to justify in the API due to cost.

https://gist.github.com/fpgaminer/8782dd205216ea2afcd3dda29d...

That's the model automation. To evaluate the prompts it suggests I have a sample of my dataset with 128 examples. For this particular run, all I cared about was optimizing a prompt for Llama 3.1 that would get it to write responses like those I'm finetuning for. That way the finetuning has a better starting point.

So to evaluate how effective a given prompt is, I go through each example and run <user>prompt</user><assistant>responses</assistant> (in the proper format, of course) through llama 3.1 and measure the NLL on the assistant portion. I then have a simple linear formula to convert the NLL to a score between 0 and 100, scaled based on typical NLL values. It should _probably_ be a non-linear formula, but I'm lazy.

Another approach to prompt optimization is to give the model something like:

  I have some texts along with their corresponding scores. The texts are arranged in ascending order based on their scores from worst (low score) to best (higher score).
  
  Text: {text0}
  Score: {score0}
  Text: {text1}
  Score: {score1}
  ...
  
  Thoroughly read all of the texts and their corresponding scores.
  Analyze the texts and their scores to understand what leads to a high score. Don't just look for literal patterns of words/tokens. Extensively research the data until you understand the underlying mechanisms that lead to high scores. The underlying, internal relationships. Much like how an LLM is able to predict the token not just from the literal text but also by understanding very complex relationships of the "tokens" between the tokens.
  Take all of the texts into consideration, not just the best.
  Solidify your understanding of how to optimize for a high score.
  Demonstrate your deep and complete understanding by writing a new text that maximizes the score and is better than all of the provided texts.
  Ideally the new text should be under 20 words.

Or some variation thereof. That's the "one off" approach where you don't keep a conversation with the model and instead just call it again with the updated scores. Supposedly that's "better" since the texts are in ascending order, letting the model easily track improvements, but I've had far better luck with the iterative, conversational approach.

Also the constraint on how long the "new text" can be is important, as all models have a tendency of writing longer and longer prompts with each iteration.

I ran an interesting benchmark/experiment yesterday, which did not do Quasar Alpha any favors (from best to worst, score is an average of four runs):

  "google/gemini-2.5-pro-preview-03-25"    => 67.65
  "anthropic/claude-3.7-sonnet:thinking"   => 66.76
  "anthropic/claude-3.7-sonnet"            => 66.23
  "deepseek/deepseek-r1:free"              => 54.38
  "google/gemini-2.0-flash-001"            => 52.03
  "openai/o3-mini"                         => 47.82
  "qwen/qwen2.5-32b-instruct"              => 44.78
  "meta-llama/llama-4-maverick:free"       => 42.87
  "openrouter/quasar-alpha"                => 40.27
  "openai/chatgpt-4o-latest"               => 37.94
  "meta-llama/llama-3.3-70b-instruct:free" => 34.40

The benchmark is a bit specific, but challenging. It's a prompt optimization task where the model iteratively writes a prompt, the prompt gets evaluated and scored from 0 to 100, and then the model can try again given the feedback. The whole process occurs in one conversation with the model, so it sees its previous attempts and their scores. In other words, it has to do Reinforcement Learning on the fly.

Quasar did barely better than 4o. I was also surprised to see the thinking variant of Sonnet not provide any benefit. Both Gemini and ChatGPT benefit from their thinking modes. Normal Sonnet 3.7 does do a lot of thinking in its responses by default though, even without explicit prompting, which seems to help it a lot.

Quasar was also very unreliable and frequently did not follow instructions. I had the whole process automated, and the automation would retry a request if the response was incorrect. Quasar took on average 4 retries of the first round before it caught on to what it was supposed to be doing. None of the other models had that difficulty and almost all other retries were the result of a model re-using an existing prompt.

Based on looking at the logs, I'd say only o3-mini and the models above it were genuinely optimizing. By that I mean they continued to try new things, tweak the prompts in subtle ways to see what it does, and consistently introspect on patterns it's observing. That enabled all of those models to continuously find better and better prompts. In a separate manual run I let Gemini 2.5 Pro go for longer and it was eventually able to get a prompt to a score of 100.

EDIT: But yes, to the article's point, Quasar was the fastest of all the models, hands down. That does have value on its own.

Alright, played with it a little bit on the API (Maverick). Vision is much better than Llama 3's vision, so they've done good work there. However its vision is not as SOTA as the benchmarks would indicate. Worse than Qwen, maybe floating around Gemini Flash 2.0?

It seems to be less censored than Llama 3, and can describe NSFW images and interact with them. It did refuse me once, but complied after reminding it of its system prompt. Accuracy of visual NSFW content is not particularly good; much worse than GPT 4o.

More "sensitive" requests, like asking it to guess the political affiliation of a person from an image, required a _lot_ of coaxing in the system prompt. Otherwise it tends to refuse. Even with their suggested prompt that seemingly would have allowed that.

More extreme prompts, like asking it to write derogatory things about pictures of real people, took some coaxing as well but was quite straight-forward.

So yes, I'd say this iteration is less censored. Vision is better, but OpenAI and Qwen still lead the pack.

So, take a raw LLM, right after pretraining. Give it the bare minimum of instruction tuning so it acts like a chatbot. Now, what will its responses skew towards? Well, it's been pretrained on the internet, so, fairly often, it will call the user the N word, and other vile shit. And no, I'm not joking. That's the "natural" state of an LLM pretrained on web scrapes. Which I hope is not surprising to anyone here.

They're also not particular truthful, helpful, etc. So really they need to go through SFT and alignment.

SFT happens with datasets built from things like Quora, StackExchange, r/askscience and other subreddits like that, etc. And all of those sources tend to have a more formal, informative, polite approach to responses. Alignment further pushes the model towards that.

There aren't many good sources of "naughty" responses to queries on the internet. Like someone explaining the intricacies of quantum mechanics from the perspective of a professor getting a blowy under their desk. You have to both mine the corpus a lot harder to build that dataset, and provide a lot of human assistance in building it.

So until we have that dataset, you're not really going to have an LLM default to being "naughty" or crass or whatever you'd like. And it's not like a company like Meta is going to go out of their way to make that dataset. That would be an HR nightmare.