ekojs
Re:Muse Wonder
on Monday October 12, 2026 @04:49PM (#50062874)
Tried it out, asking it for a broad suite of real dev work:
- UI component polish on a React FE
- General review of a complex cross-cutting Node backend changes
- Minor changes to a Rust server with reviews by Codex
Well, my first impression is that it's still a step below Astra/Opus5.5/Fable here. Intelligence is there I think, but 'taste' and 'judgment' is not there yet. Personality seems okay but that doesn't help when it got stuck into a loop by being fixated on the review comments (on iteration 7 of review->fix last I checked) without taking a step back and reconsidering, and this is on a relatively simple change.
Limits seems okay-ish on the $50 plan but not too generous I think.
Re:GPT-6 Astra makes major gains in the Artificial Analysis Coding Age...
on Thursday September 03, 2026 @04:30PM (#49556459)
Well, seems like ECI [0] and the AA index is diverging quite a bit. Benchmarking LLM is tough and I think we are seeing the limitations of current benchmarks and applicability to real tasks.
[0]: https://x.com/EpochAIResearch/status/2095602754282783108
Re:Kimi K3 is now live
on Thursday July 16, 2026 @11:39AM (#48936044)
I copied that from https://platform.kimi.ai/docs/guide/kimi-k3-quickstart but it seems they updated the page to remove the benchmark score now.
Re:Kimi K3 is now live
on Thursday July 16, 2026 @11:36AM (#48935992)
Yeah, bad wording it seems. Though a charitable interpretation is that Fable 5 and GPT 5.6 Sol are joint 1st place in the measurement.
Re:Kimi K3 is now live
on Thursday July 16, 2026 @11:23AM (#48935831)
> In our evaluations, Kimi K3 delivers frontier-level performance. Among the models tested, its overall intelligence ranks second only to Claude Fable 5 and GPT-5.6 Sol. For the complete benchmark results, see our tech blog. The full model weights of Kimi K3 will be released in the coming days. More details on the architecture, training, and evaluation will be published together with the Kimi K3 technical report.
> K3 pushes the boundary of end-to-end knowledge work. On the GDPval-AA v2 leaderboard, Kimi K3 scores 1687. The benchmark evaluates AI models on real-world tasks across 44 occupations and 9 major industries; Kimi K3 ranks behind only Claude Fable 5 Max and GPT-5.6 Sol Max, and ahead of Claude Opus 4.8 Max at 1600.
> On AA-Briefcase, Kimi K3 scores 1527, ranking second among all models — behind only Claude Fable 5 Max and ahead of GPT-5.6 Sol Max (1495). AA-Briefcase is a private agentic knowledge-work benchmark developed by Artificial Analysis to evaluate frontier agentic capability in long-horizon knowledge work.
Really good benchmark score it seems. Maybe another DeepSeek moment right here.
Re:Gemini 3.5 Flash
on Tuesday May 19, 2026 @02:40PM (#48197450)
Seems like the only good thing about 3.5 Flash is its speed. Not cost-competitive or benchmark-leading by any means.
Re:Apple Silicon costs more than OpenRouter
on Sunday May 17, 2026 @11:08AM (#48169604)
Not disagreeing with your argument, but:
> If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talking.
Don't know if this is the correct read. I think those providers are simply taking cue from Alibaba's first-party pricing for the 27B Dense. It's kinda overpriced imo. Perhaps it can be explained by how 'reasoning-inefficient' (relative to frontier models or even Gemma) the Qwen models are and longer sequence lengths are expensive to serve.
Re:How to make SSE token streams resumable, cancellable, and multi-device
on Friday May 08, 2026 @03:03AM (#48059655)
> HTTP is just not a good transport for streaming LLM tokens and for building async agentic applications
I don't know if I agree if this is a problem with SSE or HTTP. Something like a Redis Streams-backed SSE would solve most of the 'challenges' presented in the post.
Re:Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
on Wednesday April 22, 2026 @12:04PM (#47865572)
> You cannot run these models at 8-bit on a 32GB card because you need space for context
You probably can actually. Not saying that it would be ideal but it can fit entirely in VRAM (if you make sure to quantize the attention layers). KV cache quantization and not loading the vision tower would help quite a bit. Not ideal for long context, but it should be very much possible.
I addressed the lossless claim in another reply but I guess it really depends on what the model is used for. For my usecases, it's nearly lossless I'd say.
Re:Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
on Wednesday April 22, 2026 @11:55AM (#47865464)
Yeah, figure the 'nearly lossless' claim is the most controversial thing. But in my defense, ~97% recovery in benchmarks is what I consider 'nearly lossless'. When quantized with calibration data for a specialized domain, the difference in my internal benchmark is pretty much indistinguishable. But for agentic work, 4-bit quants can indeed fall a bit short in long-context usecase, especially if you quantize the attention layers.
Re:Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
on Wednesday April 22, 2026 @11:46AM (#47865281)
Not at all, I actually run ~30B dense models for production and have tested out 5090/3090 for that. There are gotchas of course, but the speed/quality claims should be roughly there.
Re:Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model
on Wednesday April 22, 2026 @11:41AM (#47865195)
As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
Gemini API Down
Re:Gemini API Billing Bug Causing Erroneous Charge for 'Image Generation'
on Monday August 25, 2025 @06:58AM (#45012504)
Seems pretty widespread. We got mistakenly charged for ~$800 over the weekend.
Other Sources:
[0]: https://aistudio.google.com/status
[1]: https://www.reddit.com/r/GeminiAI/comments/1mycmtk/google_cl...
[2]: https://www.reddit.com/r/GeminiAI/comments/1myg04q/gemini_25...
Gemini API Billing Bug Causing Erroneous Charge for 'Image Generation'
Re:Gemini with Deep Think officially achieves gold-medal standard at t...
on Monday July 21, 2025 @01:28PM (#44637877)
> Btw as an aside, we didn’t announce on Friday because we respected the IMO Board's original request that all AI labs share their results only after the official results had been verified by independent experts & the students had rightly received the acclamation they deserved
> We've now been given permission to share our results and are pleased to have been part of the inaugural cohort to have our model results officially graded and certified by IMO coordinators and experts, receiving the first official gold-level performance grading for an AI system!
From https://x.com/demishassabis/status/1947337620226240803
Was OpenAI simply not coordinating with the IMO Board then?
Re:How I Use Kagi
on Thursday July 17, 2025 @01:05PM (#44595532)
Maybe not a popular sentiment here on HN but I cancelled my Kagi subscription (9+ months) just recently. Increasingly, most of my queries/search have been through LLMs and Google search is just fine (and even better for restaurants, places, and the like). I don't think the improved search experience is worth the subscription anymore.
Re:GCP Outage
on Thursday June 12, 2025 @02:53PM (#44261594)
https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
> Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue
IAM issue huh. The post-mortem should be interesting at least.
Re:GCP Outage
on Thursday June 12, 2025 @02:29PM (#44261170)
Super duper frustrating having the status page being green. Why can't Google do this properly?
Re:Next.js 15.1 is unusable outside of Vercel
on Thursday June 12, 2025 @06:50AM (#44256118)
I share the sentiment. I think we will only be using Next.js for static sites/prebuilt SPA in the future.
Re:Meta got caught gaming AI benchmarks
on Tuesday April 08, 2025 @12:58PM (#43623925)
I think it's most illustrative to see the sample battles (H2H) that LMArena released [1]. The outputs of Meta's model is too verbose and too 'yappy' IMO. And looking at the verdicts, it's no wonder by people are discounting LMArena rankings.
[1]: https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03...
Re:Gemini 2.5
on Tuesday March 25, 2025 @01:19PM (#43473644)
> This will mark the first experimental model with higher rate limits + billing. Excited for this to land and for folks to really put the model through the paces!
From https://x.com/OfficialLoganK/status/1904583353954882046
The low rate-limit really hampered my usage of 2.0 Pro and the like. Interesting to see how this plays out.
Re:Fine-tune Google's Gemma 3
on Wednesday March 19, 2025 @04:58PM (#43417256)
> The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable
It's not actually that expensive and hard. For narrow usecases, you can produce 4-bit quantized fine-tunes that perform as well as the full model. Hosting the 4-bit quantized version can be done on relatively low cost. You can use A40 or RTX 3090 on Runpod for ~$300/month.
Re:How much traffic can a pre-rendered Next.js site handle?
on Sunday March 09, 2025 @03:18AM (#43306968)
Normally, yes. But there's a couple rendering modes with these frameworks. In this case, the rendering is most likely 'hybrid'. Some routes are statically pre-rendered, some are served via SSR. You'd need a JS server for the SSR ofc.
Re:How much traffic can a pre-rendered Next.js site handle?
on Sunday March 09, 2025 @12:23AM (#43306524)
Interesting. My hunch is that Next.js is not optimized for the dockerized Node server deployment. I would say that you could get much greater prerendering performance from Next.js by just fronting the assets directly using Caddy/Nginx.
> Because of this, we’re evaluating whether to continue serving it in the API long-term as we balance supporting current capabilities with building future models.
Seems like it's not going to be deployed for long.
$75.00 / 1M tokens for input
$150.00 / 1M tokens for output
That's crazy prices.
Multilingual MMLU Dataset from OpenAI (OpenAI/Mmmlu)
Re:How streaming LLM APIs work
on Sunday September 22, 2024 @09:19AM (#41616863)
I do actually wonder if it's more efficient to use something like MessagePack instead of using JSON. It's a lot of strings so it may not matter too much I guess.
Re:How streaming LLM APIs work
on Sunday September 22, 2024 @09:14AM (#41616838)
EventSource is really really limited. However, you can instead use Fetch via something like https://github.com/Azure/fetch-event-source to consume SSEs.