amarble

I've been getting 503s from the API. When I first checked everything was green, then it said "Claude.ai (including the desktop and mobile apps), Claude Code and Claude Cowork" and I see now they've included the API in the failure as well

I did something similar in March using Opus 4.6 (iirc) on google's "Parameter Golf" challenge, "a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte)."

I never ended up writing it up, but you can watch me spend $600 as it explored different experiments: https://github.com/rbitr/parameter-golf/blob/main/BUDGET.md

I found, similar to another comment, that it got in local minima very easily and continued to pursue loosing ideas instead of exploring (despite being prompted to do so and being aware of how much budget it had left). I also found it tended to ignore instructions. And one example, when I fed it a better solution that had come along from the public leaderboard, it ignored everything it had done and started exploring locally around that new solution, which wasn't very interesting or productive.

Would be interesting to re-run with a newer model but it's hard for me to justify the money again.

They also indicate they have a 276B A12B version, but it doesn't seem the weights are available. This might actually be able to fit in 128GB when quantized to 2 bits or so which makes it interesting.

> I believe eval startups can work when they're targeting safety benchmarks specifically.

What's changed (and was already changing before the article was written) is "safety" shifting from being mostly for show, for example confirming a LLM will refuse to "tell me how to build a bomb" or provide info found on wikipedia about drugs or weapons or whatever, to testing for (a) commercial risk and separately (b) real danger due for example to capability uplift. Both are still niche at this point, but are increasingly relevant. I think there's less demand for safety benchmarking but more the evaluations I mention.

Location: Canada

Remote: Or regular travel OK

Willing to relocate: Never say never, open to discuss

Technologies: Not a software developer. Open to interesting roles in startups or consulting. ~20 year career spanning academia, Big 4 consulting, startups.

https://marble.onl

The point of a discussion site is to hear what other people think and get different perspectives. Just getting an LLM's insightful, well-thought-out response isn't really a big draw, if one is looking for that, there's a pretty obvious way to get it. I posted this the other day (ignore the title I realized later it's too clickbaity) but this is why IMO LLMs won't replace the workforce, people aren't looking for answers to things, they're looking for other people's takes: https://news.ycombinator.com/item?id=47299988

As a counterpoint, I also tried writing something with Claude last weekend: a Google docs clone[1]. I spent $170 on Anthropic API credits, and got something that did mostly what I asked but was basically useless. It seems that for simple interfaces for which there is an exact specification, like the recent compiler and web browser examples, it's possible to write bigger projects that "work" as a demo although not in a way they'd be viable alternatives. For anything that requires taste and judgment, we've still got a long way to go. There are lots of great demos out there but few if any real examples of vibe coded (or whatever you want to call it) software standing alone as an alternative to project people wrote.

[1] https://www.marble.onl/posts/this_cost_170.html

This weekend I tried what I'd call a medium scale agentic coding project[0], following what Anthropic demonstrated last week autonomously building a C-compiler [1]. Bottom line is, it's possible to make demos that look good, but it really doesn't work well enough to build software you would actually use. This naturally lends itself to the "everybody is taking about how great it is but nobody is building anything real with it" construct we're in right now. It is great, but also not really useful.

[0] https://www.marble.onl/posts/this_cost_170.html

[1] https://www.anthropic.com/engineering/building-c-compiler

Location: Canada (Toronto)

Remote: Or hybrid, fine to regularly travel to anywhere

Willing to relocate: Would consider it long-term

I'm not a software developer, I "know how to code" but don't want any job where I'd be assessed on my code quality. ~20 years experience spanning academic research, management consulting, machine learning consulting, and AI risk assessment. My niche has been early stage companies that need someone who actually can write and understand code (scripts, not "software") but also build out a framework for how it is used, talk to customers, present results, etc. This has worked out for me in areas like evaluating and improving AI systems (computer vision and (L)LMs mostly).

Resume: https://www.marble.onl/resume_marble_Feb26.pdf

Looking for contract / freelance, especially at early stage startups

Location: Canada - travel is fine, including international and weekly for in-person

Willing to Relocate: Looking principally for contract work, I would be open to relocating for a long term opportunity.

Technologies: Most recently AI evals, past includes computer vision for manufacturing, explainability, general AI/ML research, usually business focused. I'm mostly an IC but can field or manage a team as required to deliver AI projects across a broad range of specialties.

Resume: https://www.marble.onl/resume_nov_2025.pdf

Looking for contract / freelance

Location: Canada - travel is fine

Technologies: Most recently AI evals, past includes computer vision for manufacturing, explainability, general AI/ML research, usually business focused. I'm mostly an IC but can field a team as required to deliver AI projects across a broad range of specialties.

Resume: https://www.marble.onl/resume_nov_2025.pdf

Looking for contract / freelance

Location: Canada - travel is fine

Technologies: Most recently AI evals, past includes computer vision for manufacturing, explainability, general AI/ML research, usually business focused

Resume: https://www.marble.onl/resume_nov_2025.pdf

Since we're sharing, I have a "claude" command that lets me get quick answers but also saves the conversation and outputs an identifier so in the rare case I want a follow-up, I can ask a question with the ID to continue the conversation.

https://gist.github.com/rbitr/bfbc43b806ac62a5230555582d63d4...