News for hackers. Stuff that matters.
Re:Our next-generation model: Gemini 1.5 by alphabetting on Thursday February 15, 2024 @10:26AM (#39383799)
Massive whoa if true from technical report

"Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens"

https://storage.googleapis.com/deepmind-media/gemini/gemini_...

Re:Our next-generation model: Gemini 1.5 by Workaccount2 on Thursday February 15, 2024 @10:48AM (#39384145)
10M tokens is absolutely jaw dropping. For reference, this is approximately thirty books of 500 pages each.

Having 99% retrieval is nuts too. Models tend to unwind pretty badly as the context (tokens) grows.

Put these together and you are getting into the territory of dumping all your company documents, or all your departments documents into a single GPT (or whatever google will call it) and everyone working with that. Wild.

Re:Our next-generation model: Gemini 1.5 by kranke155 on Thursday February 15, 2024 @11:22AM (#39384593)
Seems like Google caught up. Demis is again showing an incredible ability to lead a team to make groundbreaking work.
Re:Our next-generation model: Gemini 1.5 by huytersd on Thursday February 15, 2024 @12:15PM (#39385312)
If any of this is remotely true, not only did it catch up, it’s wiping the floor with how useful it can be compared to GPT4. Not going to make a judgement until I can actually try it out though.
Re:Our next-generation model: Gemini 1.5 by singularity2001 on Thursday February 15, 2024 @01:22PM (#39386265)
In the demo videos gemini needs about a minute to answer long context questions. Which is better than reading thousands of pages yourself. But if it has to compete with classical search and skimming it might need some optimization.
Re:Our next-generation model: Gemini 1.5 by a_wild_dandan on Thursday February 15, 2024 @04:19PM (#39389034)
Replacing grep or `ctrl+F` with Gemini would be the user's fault, not Gemini's. If classical search for a job already a performant solution, use classical search. Save your tokens for jobs worthy of solving with a general intelligence!
Re:Our next-generation model: Gemini 1.5 by Breza on Tuesday February 20, 2024 @02:18PM (#39445593)
I think some of the most useful apps will involve combining this level of AI with traditional algorithms. I've written lots of code using the OpenAI APIs and I look forward to seeing what can be done here. If you type, "How has management's approach to comp changed over the past five years?" it would be neat to see an app generate the greps needed to find the appropriate documents and then feed them back into the LLM to answer the question.
Re:Our next-generation model: Gemini 1.5 by huytersd on Thursday February 15, 2024 @02:27PM (#39387282)
That’s a compute problem, something that involves just throwing money at the problem.
Re:Our next-generation model: Gemini 1.5 by IanCal on Friday February 16, 2024 @05:53AM (#39395408)
If you had this for your business could this approach be faster than RAG?

Input is parsed one token at a time right? Can you cache the state after the initial prompt has been provided?

Re:Our next-generation model: Gemini 1.5 by og_kalu on Thursday February 15, 2024 @11:16AM (#39384521)
Another whoa for me

>Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.

Results - https://imgur.com/a/qXcVNOM

Re:Our next-generation model: Gemini 1.5 by usaar333 on Thursday February 15, 2024 @11:49AM (#39384999)
I think this somewhat is mostly due to the ability to handle high context lengths better. Note how Claude 2.1 already highly outperforms GPT-4 on this task.
Re:Our next-generation model: Gemini 1.5 by a_wild_dandan on Thursday February 15, 2024 @04:23PM (#39389100)
GPT-4V turbo outperforms Claude on long contexts, IIRC. Unless that's mistaken, I'd suspect a different explanation for that task.
Re:Our next-generation model: Gemini 1.5 by stavros on Thursday February 15, 2024 @10:31AM (#39383873)
Until I can talk to it, I care exactly zero.
Re:Our next-generation model: Gemini 1.5 by peterisza on Thursday February 15, 2024 @11:25AM (#39384656)
you can buy their stock if you think they'll make a lot of money with their tech
Re:Our next-generation model: Gemini 1.5 by HarHarVeryFunny on Thursday February 15, 2024 @12:02PM (#39385140)
Well that's really the right question .. what can, and will, Google do with this that can move their corporate earnings needle in a meaningful way? Obviously they can sell API access and integrate it into their Google docs suite, as well as their new Project IDX IDE, but do any of these have potential to make a meaningful impact ?

It's also not obvious how these huge models will fare against increasingly capable open source ones like Mixtral, perhaps especially since Google are confirming here that MoE is the path forward, which perhaps helps limit how big these models need to be.

Re:Our next-generation model: Gemini 1.5 by plaidfuji on Thursday February 15, 2024 @12:55PM (#39385890)
In the long run it could move the needle in enterprise market share of Workspace and GCP. They have a lot of room to grow and IMO have a far superior product to O365/Azure which could be exacerbated by strong AI products. Only problem is this sales cycle can take a decade or more, and Google hasn’t historically been patient or strategic about things like this.
Re:Our next-generation model: Gemini 1.5 by megaman821 on Thursday February 15, 2024 @10:33AM (#39383891)
So, will this outperform any RAG approach as long as the data fits inside the context window?
Re:Our next-generation model: Gemini 1.5 by CuriouslyC on Thursday February 15, 2024 @11:02AM (#39384322)
A perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.
Re:Our next-generation model: Gemini 1.5 by TheGeminon on Thursday February 15, 2024 @10:40AM (#39384006)
Outperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation.

But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.

Re:Our next-generation model: Gemini 1.5 by megaman821 on Thursday February 15, 2024 @10:48AM (#39384140)
Doesn't their needle/haystack benchmark seem to suggest there is almost no dilution? They pushed that demo out to 10M tokens.
Re:Our next-generation model: Gemini 1.5 by ArcaneMoose on Thursday February 15, 2024 @10:37AM (#39383950)
Cost would still be a big concern
Re:Our next-generation model: Gemini 1.5 by saliagato on Thursday February 15, 2024 @10:38AM (#39383966)
basically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.
Re:Our next-generation model: Gemini 1.5 by _boffin_ on Thursday February 15, 2024 @10:41AM (#39384027)
Prompt token cost still a variable.
Re:Our next-generation model: Gemini 1.5 by chasd00 on Thursday February 15, 2024 @04:22PM (#39389091)
are you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.
Re:Our next-generation model: Gemini 1.5 by cchance on Thursday February 15, 2024 @12:36PM (#39385590)
Did you watch the video of the Gemini 1.5 video recall after it processed the 44 minute video... holy shit
Re:Our next-generation model: Gemini 1.5 by matsemann on Thursday February 15, 2024 @10:58AM (#39384278)
Could you (or someone) explain what this means?
Re:Our next-generation model: Gemini 1.5 by FergusArgyll on Thursday February 15, 2024 @12:36PM (#39385581)
The input you give it can be very long. This can qualitatively change the experience. Imagine, for example, copy pasting the entire lord of the rings plus another 100 books you like and asking it to write a similar book...
Re:Our next-generation model: Gemini 1.5 by HarHarVeryFunny on Thursday February 15, 2024 @01:07PM (#39386068)
I just googled it, and the LOTR trilogy apparently has a total of 480,000 words, which brings home how huge 1M is! It'd be fascinating to see how well Gemini could summarize the plot or reason about it.

One point I'm unclear on is how these huge context sizes are implemented by the various models. Are any of them the actual raw "width of the model" that is propagated through it, or are these all hierarchical summarization and chunk embedding index lookup type tricks?

Re:Our next-generation model: Gemini 1.5 by mburns on Thursday February 15, 2024 @04:41PM (#39389377)
For another reference, Shakespeare’s complete works are ~885k words.

The Encyclopedia Britannica is ~44M words.

Re:Our next-generation model: Gemini 1.5 by teaearlgraycold on Thursday February 15, 2024 @12:54PM (#39385886)
I doubt it’s smart enough to write another (coherent, good) book based on 103 books. But you could ask it questions about the books and it would search and synthesize good answers.
Re:Our next-generation model: Gemini 1.5 by staticman2 on Thursday February 15, 2024 @05:36PM (#39390194)
Reading Lord of the Rings, and writing a quality book in the same style, are almost wholly unrelated tasks. Over 150 million copies of Lord of the Rings have been sold, but few readers are capable of "writing a similar book" in terms of quality. There's no reason to think this would work well.
Re:Our next-generation model: Gemini 1.5 by pfooti on Thursday February 15, 2024 @09:01PM (#39392184)
I mean, Terry Brooks did it with the Sword of Shannara. (/s)
Re:Our next-generation model: Gemini 1.5 by ehsankia on Thursday February 15, 2024 @02:38PM (#39387463)
It's how much text it can consider at a time when generating a response. Basically the size of the prompt. A token is not quite a word but you can think of it as roughly that. Previously, the best most LLMs could do is around 32K. This new model does 1M, and in testing they could put it up to 10M with near perfect retrieval.

As the other comment mentions, you can paste the content of entire books or documents and ask very pointed question about it. Last year, Anthropic was showing off their 100K context window, and that's exactly what they did, they gave it the content of The Great Gatsby and asked it questions about specific lines of the book.

Similarly, imagine giving it hundreds of documents and asking it to spot some specific detail in there.

Re:Our next-generation model: Gemini 1.5 by liamYC on Thursday February 15, 2024 @06:31PM (#39390797)
Awesome explanation, thanks for the comparison
Re:Our next-generation model: Gemini 1.5 by Breza on Tuesday February 20, 2024 @02:20PM (#39445612)
Great explanation. I was amazed when I started using Claude because I could find a recently-transcribed novella, upload it, and ask specific questions. I'm downright giddy to try a 1M+ model.