platypii

Yea it's been fun! Everyone pretty much assumed n log n was the actual limit for integer multiplication. So the OpenAI result was surprising!

But of the 722 proofs openai released, integer multiplication did NOT come with a lean proof. So I wanted to see if the result was real, or maybe was a flaw in the proof. So I created the repo, and started burning tokens. It was not easy even with AI! It took a couple days laying the ground work. A huge part of the challenge was keeping the proof checking fast. It took ~8 hours for lean to check the first version of the proof. I fought to get it back down to ~20 minutes. Then huge credit to @jdb1729 for joining the effort and actually completing the proof across the finish line! The theorem was true!

Once we proved the initial openai bound of 2^-182, we've been working on integrating improvements to the bound, and we're now up to K = 2^-14. Continuing to work on it, community contributions welcome!

Sylvain Lesage’s cool interactive explainer on visualizing extreme row counts—think billions—inside the browser. His technical deep dive explains how the open-source library HighTable works around scrollbar limits by:

- Lazy loading - Virtual scrolling (allows millions of rows) - "Infinite Pixel Technique" (allows billions of rows)

Hyperparam sponsored Sylvain’s work as part of our broader effort to invest in open-source infrastructure and get ahead of the data-scale problems that are emerging with LLMs. With a regular table, you can view thousands of rows, but the browser breaks pretty quickly. We created HighTable with virtual scroll so you can see millions of rows, but that still wasn’t enough for massive unstructured datasets. What Sylvain has built virtualizes the virtual scroll so you can literally view billions of rows—all inside the browser. His write-up goes deep into the mechanics of building a ridiculously large-scale table component in react.

We're willing to spend money, but I've had the "datadog billing problem" before where it starts reasonable and then grows to a non-trivial percent of saas budget, and then theres a scramble to refactor. Trying to get ahead of that as the LLM logs are MUCH larger that my APM logs.

Makes sense. I'm not currently in snowflake because I'm mostly working with local parquet files. Would prefer not to have to pay for snowflake just to explore my data. I'm interested in better data UIs though so I might need to check it out.

I started Hyperparam one year ago because I knew that the world of data was changing, and existing tools like Python and Jupyter Notebooks were not built for the scale of LLM data. The weights of LLMs may be tensors, but the input and output of LLMs are massive piles of text.

No human has the patience to sift through all that text, so we need better tools to help us understand and analyze it. That's why I built Hyperparam to be the first tool specifically designed for working with LLM data at scale. No one else seemed to be solving this problem.

This is a Q&A I did on what I learned from a year of open source data transformation. Most of all, it reinforced my belief that browser-native tools aren’t “toys” that don’t work for real systems. When Hugging Face integrated my libraries, it confirmed that the browser can handle serious data work, and maybe there's an opportunity for more browser-based data tools.

As with anything, there are engineering tradeoffs.

What I've found is that moving data processing toward the browser has been for one, a refreshing developer experience because I don't need to build a pair of backend+frontend. From a user experience point of view, I think you can build MORE interactive data applications by pushing it toward the frontend.

Why not? We are trying to evaluate AI's capabilities. It's OBVIOUS that we should compare it to our only prior example of intelligence -- humans. Saying we shouldn't compare or anthropomorphize machine is a ridiculous hill to die on.

This is the story of how I spent a year making the world's fastest Parquet loader in JavaScript. The goal:

- Make a faster, more interactive viewer for AI datasets (which are mostly parquet format)

- Simplify the stack by doing everything from the browser (no backend)

TLDR: My open-source library Hyparquet can load data in 155ms, which would take 3466ms in duckdb-wasm for the same file.

I don’t have benchmarks specifically against duckdb. I’m sure native C++ will run faster than JavaScript.

But whats important is that with Hyperparam you can do it in the browser, where the bottleneck will always be network-bound not cpu-bound.

Funny you say that, because I built these tools because I wanted to build something very much like what you're describing!

I was trying to look at, filter, and transform large AI datasets, and I was frustrated with how bad the existing tool was for working with datasets with huge amounts of text (web scrapes, github dumps, reasoning tokens, agent chat logs). Jupyter notebook is woefully bad at helping you to look at your data.

So I wanted to build better browser tools for working with AI datasets. But to do that I had to build these tools (there was no working parquet implementation in JS when I started).

Anyway I'm still working on building an app for data processing using LLM chat assistant to help a single user curate entire datasets singlehandedly. But for now I'm releasing these components to the community as open source. And having them "do a single task each" was very much intentional. Thanks for the comment!

Yea except with parquet you don't need to load the entire file, the parquet metadata let's you do http range requests for just the data you need.

For example this parquet is the entire english wikipedia (400mb) but loads less than 4mb including html and all js to display the first rows:

https://hyperparam.app/files?key=https%3A%2F%2Fs3.hyperparam...

This way you can have huge AI datasets in cloud storage, and still have a nice interface for looking at your data.

In particular, a lot of modern AI datasets are huge walls of text (web scrapes, chains of thought, or agentic conversation histories), and most datasets on huggingface are in parquet. So you can much more quickly look at your data this way versus say jupyter notebooks.

Here's the glaive reasoning dataset on the Hyperparam hugging face space:

https://huggingface.co/spaces/hyperparam/hyperparam?url=http...

That's fair criticism... to be honest when I started the project it was more focused on hyperparameters, and it evolved into this javascript-for-ai mission. But now I just kind of liked the name.

It does support using S3 presigned requests, but it's admittedly a little awkward to ask a server for a presigned request before every fetch. But does still have the benefit that you can have a small and light server just handing out signed requests, and then the user and their browser does the heavy lifting. This can save a lot on scaling out server costs.

That being said, I wish there was a better auth story. Open to suggestions if anyone has ideas!

Duckdb and datafusion are super cool! But they are VERY large wasm blobs (30-40mb each). This is often larger than the data you’re trying to load. And they add complexity with serving and deploying wasm files.

Hyparquet is 10kb of pure js, and so its trivial to deploy on a modern webapp, and wins hands down on time-to-first-data metric.

Zero telemetry, fully local. It spawns `http-server` on port 2048 and opens your browser at `localhost`. Similar pattern as Jupyter Notebooks. Feel free to audit the code... the server is <200 LOC.

FSD makes Tesla superior to any car out there. No other car comes even close.

Although I heard that FSD was already crippled in eu so maybe they aren't missing out as much.

AI is even better at writing tests than writing code. So have it write the tests first and then write the code.

My goal is to build tools which enable working with large-scale ML datasets in the browser. The browser is critical for building compelling UIs, but previous parquet js libraries had gone abandoned.

Apache Parquet is a very complicated format. It has 22 data types, 9 encodings, 8 compression codecs. However, I can confidently say that Hyparquet is now the most conformant parquet parser in existence. It can open all the parquet files: more than PyArrow and DuckDB. I dare you to find a file that Hyparquet can’t open!

Hyparquet is MIT licensed, and there is a demo github page which can open parquet files in the browser with no backend server.