hansonw

The skill source is here: https://github.com/openai/skills/blob/main/skills/.curated/p...

$skill-installer playwright-interactive in Codex! the model writes normal JS playwright code in a Node REPL

Rest assured that we are better at training models than naming them ;D

- New benchmark SOTAs with 77.9% on SWE-Bench-Verified, 79.9% on SWE-Lancer, and 58.1% on TerminalBench 2.0

- Natively trained to work across many hours across multiple context windows via compaction

- 30% more token-efficient at the same reasoning level across many tasks

Let us know what you think!

More about that here! https://platform.openai.com/docs/codex#advanced-configuratio...

The ELI5 of the paper is that most "unlearning" methods can be regarded as adding some delta `w` to the parameters of the network, but most of `w` just gets "rounded away" during quantization (i.e. `quantize(X+w) ~= quantize(X)`). Pretty clever idea as a lot of cited methods explicitly optimize/regularize to keep `w` small to avoid degrading evaluation accuracy.

To your point, it does put into question the idea of whether these methods can actually be considered truly "unlearning" from an information-theoretic perspective (or if it is the equivalent of e.g. just putting `if (false)` around the still latent knowledge)

It looks like they didn't want to make a public submission in order to avoid disclosing the model internals: https://cosine.sh/blog/genie-technical-report#:~:text=SWE%2D....

It probably actually is several GB / 1M tokens. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1M 16-bit floats per token = 2GB

This is also a good paper on the subject:

What Algorithms can Transformers Learn? A Study in Length Generalization https://arxiv.org/abs/2310.16028

Yes. But also note that the new function calling is actually “tool calling” where the model is also fine-tuned to expect and react to the output of the function (and there are various other nuances like being able to call multiple functions in parallel and matching up the outputs to function calls precisely).

When used in multi-turn “call/response” mode it actually does start to unlock some new capabilities.

Not the author, but really nice that they shared some real data points:

> Once our Text-to-SQL solution was in production, we were also able to observe how users interacted with the system. As our implementation improved and as users became more familiar with the feature, our first-shot acceptance rate for the generated SQL increased from 20% to above 40%. In practice, most queries that are generated require multiple iterations of human or AI generation before being finalized. In order to determine how Text-to-SQL affected data user productivity, the most reliable method would have been to experiment. Using such a method, previous research has found that AI assistance improved task completion speed by over 50%. In our real world data (which importantly does not control for differences in tasks), we find a 35% improvement in task completion speed for writing SQL queries using AI assistance.

If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)

If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).

Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!

“RNN-mode inference” is also extremely exciting because you can precompute the hidden state of any prompt prefix (i.e. a long system prompt, or statically retrieved context) and continued generations pay the same cost irrespective of the prefix length.

Our startup is building https://arcwise.app, which allows you to embed full-fledged SQL tables inside Google Sheets! We’re in the process of building out support for joins & subqueries, would be curious what people think.

I’m building a solution that works like this - we directly connect spreadsheet models to company databases (even converting pivots/formulas to SQL). Would love to chat with anyone that might find this valuable: https://arcwise.app

This is the best comparison I've found that benchmarks the current OSS inference solutions: https://hamel.dev/notes/llm/inference/03_inference.html

IME the streaming API in text-generation-inference works fine in production. (Though some of the other solutions may be better). I've used it with Starcoder (15B) and the time-to-first-token / tokens per second all seem quite reasonable out of the box.

gcsfuse has controllable built-in caching of _metadata_ but not contents: https://cloud.google.com/storage/docs/gcsfuse-performance-an...

You'd have to use your own cache otherwise. IME the OS-level page cache is actually quite effective at caching reads and seems to work out of the box with gcsfuse.

What about datasets like https://huggingface.co/datasets/b-mc2/sql-create-context? An OSS model based on StarCoder was also recently published which is roughly between GPT-3.5 and GPT-4: https://github.com/defog-ai/sqlcoder

Arcwise | https://arcwise.app | Engineering | Full-time | SF or Remote (US)

At Arcwise, we’re on a mission to make analytics and data science accessible to anybody - starting with spreadsheet users. We're big believers that the spreadsheet interface is still unmatched when working with tabular data, but spreadsheets just haven't evolved to match the rapid pace of the modern data world. To fix that, we're building a platform to supercharge the spreadsheet to keep up with today's business needs.

We're a seed-stage startup that just came out of Sequoia's Arc Americas program and are looking to hire a founding full-stack engineer; the founders were previously leaders at Flexport and Meta across business, engineering, and data science. Our tech stack is currently TypeScript/React + Python + Terraform and we're working with the latest advancements in the modern data stack (e.g. dbt's new semantic layer).

For more information, see https://hiring.arcwise.app or email your CV to [email protected]

It appears the indexing for the model parts is deliberately not contiguous; the 03-82 range represents the main 80 transformer layers. https://github.com/yandex/YaLM-100B/blob/main/megatron_lm/me...

(Note: I worked on https://ide.atom.io and Facebook's Nuclide team).

On the topic of performance - one of the architecture decisions that VSCode nailed was the 'extension host' (https://code.visualstudio.com/api/advanced-topics/extension-...): all VSCode extensions are sandboxed in a child Node process, and can only communicate back with the main IDE process via the VSCode API. This has huge benefits for performance:

(1) extensions can never block editor startup. this was a huge problem for Atom as package loads were startup blocking and it wasn't uncommon to have multi-second startup times with lots of packages installed - especially due to the fact that JS packages typically repeat dependencies, resulting in tons of bloat. Also extension authors are rarely performance-conscious

(2) extension code can never block the core rendering thread, another huge problem in Atom - you'd often have stuttering/frame drops if extensions were doing blocking work on character or cursor changes, which was more often the case than not..

The tradeoff of course is that VSCode extensions are very limited in the set of UI customizations they can make but MS did a very good job of designing their APIs to be sufficiently extensible in this aspect (i.e. having APIs for extensions to hook into all core IDE features). Atom's extension ecosystem was much more fragmented resulting in dependency/versioning hell.

As a side note, another benefit of the extension host model is how it enables extensions to semi-magically work on remote filesystems (including inside WSL) without needing complete rewrites.

a bunch of Parquet files in S3 ;)

plug in https://www.etebase.com/ and ship it :D

PyScript is built on top of Pyodide! But it's probably the first well-supported way to build frontend apps with it. I imagine the fact that Anaconda is backing PyScript means that there will be increased investment into Pyodide as well.

Here's a ~5 minute talk with 5 such examples (where relatively simple ML models made a 1M+ impact at a FAANG) :) https://youtu.be/zyOEOd1HkSY?t=946 Happy to talk about more details if you message me through my profile!