mlaskin

Launching Reflection. Our team pioneered major advances in RL and LLMs, including AlphaGo, AlphaZero, and Gemini. We're building superintelligent autonomous systems. Starting with autonomous coding.

We believe that solving autonomous coding will enable superintelligence more broadly.

website: https://mishalaskin.github.io/curl/ code: https://github.com/MishaLaskin/curl

Thank you! That's a great suggestion

Definitely thanks for asking!

Traditional OCRs are good at transcribing typed notes (e.g. pdfs) to editable docs, but do poorly with handwriting. The best OCRs I've seen can make a handwritten note searchable (e.g. Evernote) but still don't transcribe it editable form.

A lot of academic work on transcribing images of handwritten notes into text has surfaced over the last couple of years (mostly regarding using neural networks), and we decided to apply it

Fair point :)

Thanks!

I was at LensCrafters the other day and had to fill out a paper form that someone input into a computer by hand, so definitely see the need there.

Our goal is to get a high enough accuracy for handwriting OCR to work in enterprise settings. 90% may be good for consumers, but I wouldn't want to put anyone's health on the line due to a transcription error

We'll do our best ;)

Thanks for the catch! We uploaded an updated version of the video here

https://youtu.be/gYPWjYDsRyk

Thanks Brad! It was a great call out and something we should've seen before posting. Appreciate the feedback

Hey! We wanted to show how the product works if notes are delivered instantly - vs. the current 10-30 sec lag time we experience between note being snapped, and the transcribed copy you receive.

Definitely didn’t mean for it to put a damper on the tech - sorry for the mistake.

thanks for the question!

It handles that kind of script pretty well, mainly because we had a good amount of training examples from college student handwriting

Thank you for the question!

Currently, it's a combination of the two, mainly because people often take notes hastily so word-based recognition coupled with spell check allow you to fix things on the fly. However, this also results in bizarre outputs sometimes so we're still figuring out what an optimal output looks like.

Thanks for commenting!

Adding digital handwriting support is a great idea. I actually think other companies do it pretty well, which is why we didn't go down that route. The reason is that they use a different type of algorithm that learns, in part, from the handwriting velocity, and gives you edit access on the go, which is not possible if you've taken the notes in a normal notebook.

We decided to start with plain notebook text mainly because it seemed like no one else had solved this problem to our satisfaction yet.

Good call out, we'll edit the video description

Yep, that's fair.

We staged the google doc bc there is still a 10-30 sec lag time in sending the note out, we'll fix it up. Thank you for the call out!

Thank you for commenting!

ML / NN have been around for a while, but there are a few reasons Scribble is only possible now:

1) Although classifying MNIST digits is the "hello world" of ML, doing the same with notes is substantially more difficult. The algorithm has to figure out sentence structure, punctuation, paragraph breaks, lists, and tons of other features that are hard to train. This problem is still a major research topic academically.

2) As a corollary to (1), while OCR has been around for a while, handwriting OCR has never worked due to (1).

3) Computing power has never been so cheap, training the algorithm would have been very expensive before AWS / Azure / etc abstracted hardware and made it inexpensive

Thank you! Really appreciate your support

Scribble doesn't transcribe hand drawn figures yet.

I used to be a physics student and always had to draw plots / equations by hand, so it's definitely a feature I would love to see as well

great suggestion! We love Rocketbook

not yet

Thank you for the comment!

We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.

We did two things to train it (1) scraped the web for photographs of handwritten notes with known transcription to build our training dataset (2) had our university friends / students write out training examples by hand to get more realistic data on what modern handwriting looks like

Scribble currently only supports English, so it does poorly with other languages, but is pretty robust to poor handwriting in English (such as my own).

It gets about 85% of my handwriting correct (my handwriting is abysmal), so there's definitely room for improvement.

Thank you for the comment!

We don't have a developer facing API at the moment but it's in the roadmap. Once our algorithm is accurate enough that it "just works" in an enterprise setting, we may open up an API so developers can build applications for their businesses.

That's a great question.

It currently happens in the cloud (purely a software design decision), but we could probably do it locally as well since the NN is already trained.

We were thinking of having similar TOS as normal note editing software (Evernote, Dropbox Paper) to mitigate security concerns. What do you think?

Thanks for the comment! We had the same use case when we decided to build it.

It's currently not perfect but handles a surprising amount of bizarre handwriting styles (cursive / messy notes). Looking forward to hearing your thoughts as we onboard to the beta,

Thanks!

Agreed re: Evernote. I actually really like that feature, because it makes handwritten notes searchable but found the same problem you identified with the lack of transcription.

My hope is we can integrate with players like Evernote / OneNote who already do a great job at centralizing notes.

Thanks for the comment!

The algorithm works on a large variety of handwriting (currently only support English).

It's a NN that was trained on ~100k different handwriting examples, and it's pretty robust to cursive / sloppy handwriting. Haven't tried European vs American numbers yet but I'll definitely give it a try now that you've piqued my curiosity