Usually I find "most watched" ends up just being the most showy part of a clip, but I'm usually wanting the most informational stuff which isn't the same. E.g. I'm a rock climber and watch videos on technique and safety, but most watched is climbing a route with some pop song in the background. Useless.

One thing it's useful for is skipping past advertising segments.

How do you know what questions to ask if you haven't read the book

If you click the emoji there are some sample questions for each book.

I'd probably start with: "What is the main argument of the book?" if it's non-fiction.

Perhaps even more exciting - you could write the documentation via a conversation with the chatbot, and it could ask further questions when it is unsure how to answer a user's query and even update the documentation when changes are noticed.

Hey! I'm working on building this vision at fragen.co.uk - which can take in Youtube videos (trascribed by whisper + postprocessed into useful chunks), PDFs (properly OCR'd - understands bullet points etc) and webpages (also OCR'd, experimental) - For the problem of data-supported AI search, the content really matters. Our edge is semantically chunking it, in other words we are splitting it up into key facts ready for recall. That makes this problem more solveable.

... and our tool "sees" the above text as ...

For the problem of data-supported AI search, the content really matters. fragen.co.uk's edge is semantically chunking the content, in other words we are splitting the content up into key facts ready for recall. Splitting the content up into key facts ready for recall makes data-supported AI search solveable.

(hope it's visible how an LLM like GPT able to use/quote the above can perform seriously better at those bothersome it/what/where questions and follow ups)

Agree. I suspect that this is the short/medium term application for this tech.

One basic but effective demonstration I’ve seen was summarising a 30 minute talk on YouTube into dot points.[1]

I watched the video and read the summary afterwards and was almost completely satisfied with the summary.

At scale, the flexible compression and expansion and navigation of information is potentially huge … like Google Maps for the internet.

[1]: https://gist.github.com/simonw/9932c6f10e241cfa6b19a4e08b283...

Google Maps for human knowledge! That's such an apt analogy(?) right there. Thanks for sharing!

If you sprinkle in a bit of infrastructure, I think we're already there. The ability to distill a variety of content into vectors and perform approximate nearest neighbor search (shameless plug: https://milvus.io) across all of them can really help power a lot of these applications. With the retrieved vectors, you could match questions with answers or create a reverse index to the original content to perform summarization.

With that being said, one of the main challenges ahead will be multimodal learning. We're sort-of there combining text with visual data, but there are many other modalities out there as well.

How usable would the document embedding be for a nearest neighbor search if the dimensions were reduced to three?

Might be usable for low column count tabular data, but it would be pretty terrible for any other semantically dense modality e.g. video, molecules, geospatial, etc.

Nearly useless for most applications unless there's been a major improvement in the SOTA that I missed.

can i use the three dimensions to encode a space-filling curve over a 1000-dimensional embedding?

Not precisely, but if you had 50 documents in that 1000-dimensional embedding and you reduced the dimensions to three and still got at least the exact same nearest neighbor ordering then it would at least still function, right?

I guess the problem is taking a new document (like a search term) in the higher dimensional embedding and reducing it to three dimensions for searching in that reduced space and expecting that to also maintain the same nearest neighbor ordering.

Imagine listening to a book on tape, like even scifi/fantasy and being like wait, what just happened? Or it kinda acting like the Grandpa (minus humor) in Princess Bride, basically someone who's read the book and can stop/replay bits especially via audio. This would be a godsend for people w/ ADHD, or auditory processing disorders.

I have a decent sized book collection, and what I'd like the most is a ctrl-f for books. Does that exist already in e-readers?

Personally I think question answering is still very gimmicky, in particular because I can't clearly understand why the answer is what it is. Ctrl-f is completely explainable, I know why it works and why it fails, and is a much more useful tool to engage with books. The main problem seems that the full text of most books is not available unencumbered to process as one sees fit

Agreed. That's why the approach we are taking is to answer the question 'and' show the links/paragraphs which answer the question.

Demo we have created for our website: https://speech-kws.ozonetel.com/ozosearch

I tend to download PDFs for an physical book I buy, but I’d like a hybrid approach. Ctrl-F is perfect for finding excerpts when you have a specific wording, but not great for finding something worded differently or something in a section you have not read yet. AI could step in for those and not only give specific references from the book (or it’s own referenced material) but could also give the text in full.

“AI, what does this book say about integrity?”

“(answer), based on paragraphs on page 143 and page 210”

“Show me those paragraphs inline, starting with the paragraph where the author transitions in to the topic”

Not sure about e-readers. But any EPUB can be searched like any other text.

Yes but also no. If you have an exact string query that'll turn up the topic you want then, sure, you can find a topic easily. If your question is more tangential, say you kind of know what topic you're looking for but don't know what it's called in language X's standard library, you're pretty basically SOL unless an author happened to mention a phrase you can match.

Using chat GPT to solve this problem seems pretty awesome, it would be super handy to query my (larger than it has any right to be) library of reference books using not-exactly-competent requests.

This and translating technical material into 5th grade reading level explanations are probably the best two uses of chat GPT I've seen yet.

eg. https://huberman.rile.yt/ based on popular The Huberman Lab podcast.

imagine feeding it some person forum history, so you can talk to them even when they are dead, public figure or a family member.

A related thing I’ve been wondering about — I have thousands of bookmarks and saved PDFs etc and I would love to have a way to scrape all the content from the URLs and PDFs and then ask questions and get answers along with links to relevant docs. Now that would be a killer app.

EDIT — answered my own question: there is indeed an OpenAI Embeddings API :

https://beta.openai.com/docs/guides/embeddings

That plus a vector similarity engine (FAISS for example) is the key to these types of apps (Thanks to Simon Willison’s blog, which he pointed to elsewhere in this thread)