News for hackers. Stuff that matters.
Re:New OpenAI Feature: Predicted Outputs by cbhl on Monday November 04, 2024 @10:38PM (#42048399)
If you use the Cursor IDE: the folks that wrote it talked about their use of speculative decoding to make "Apply" faster on the Lex Friedman podcast last month.

Here it is on YouTube, although you can also find it on Spotify and other podcast platforms:

https://youtu.be/oFfVt3S51T4?t=1206

Re:New OpenAI Feature: Predicted Outputs by afro88 on Tuesday November 05, 2024 @12:14AM (#42048776)
For those that prefer text: it seems they use a weaker but faster model for the "predicted output" / speculation. Pretty smart.
Re:New OpenAI Feature: Predicted Outputs by pandada8 on Tuesday November 05, 2024 @01:32AM (#42049090)
https://fireworks.ai/blog/cursor Fireworks AI has a blog about it.
Re:New OpenAI Feature: Predicted Outputs by creativenolo on Tuesday November 05, 2024 @04:20AM (#42049954)
I found the OpenAI page to be more interesting https://platform.openai.com/docs/guides/latency-optimization...
Re:New OpenAI Feature: Predicted Outputs by msp26 on Tuesday November 05, 2024 @07:18AM (#42050922)
It's incredibly well written. I can see this being very helpful for newcomers.

As for the Predicted Outputs feature, it looks incredibly useful in a few of my pipelines. Can't wait to test it out.

Re:New OpenAI Feature: Predicted Outputs by nunez on Tuesday November 05, 2024 @09:14AM (#42051598)
This is like the likely() and unlikely() macros in the Linux kernel! Huge speedup if you're right; small penalty if you're not.
Re:New OpenAI Feature: Predicted Outputs by digdugdirk on Tuesday November 05, 2024 @03:00PM (#42054684)
Any recommendations for high level overview/learning resources about this? It seems interesting, but like most Linux internals, things get real technical, real quick.