Re:Large Enough – Mistral AI
by freediver on Wednesday July 24, 2024 @12:45PM
(#41058941)
That would make each API call cost at least $3 ($3 is price per million input tokens). And if you have a 10 message interaction you are looking at $30+ for the interaction. Is that what you would expect?
[ Parent
| on HN ]
Gemini 1.5 Pro charges $0.35/million tokens up to the first million tokens or $0.70/million tokens for prompts longer than one million tokens, and it supports a multi-million token context window.
Substantially cheaper than $3/million, but I guess Anthropic’s prices are higher.
It is also much worse.
Is it, though? In my limited tests, Gemini 1.5 Pro (through the API) is very good at tasks involving long context comprehension.
Google's user-facing implementations of Gemini are pretty consistently bad when I try them out, so I understand why people might have a bad impression about the underlying Gemini models.
Re:Large Enough – Mistral AI
by reitzensteinm on Wednesday July 24, 2024 @06:50PM
(#41063050)
You're looking at the pricing for Gemini 1.5 Flash. Pro is $3.50 for <128k tokens, else $7.
Ah... oops. For some reason, that page isn't rendering properly on my browser: https://imgur.com/a/XLFBPMI
When I glanced at the pricing earlier, I didn't notice there was a dropdown at all.
Maybe they're summarizing/processing the documents in a specific format instead of chatting? If they needed chat, might be easier to build using RAG?
This might be when it's better to not use the API and just pay for the flat-rate subscription.
Re:Large Enough – Mistral AI
by impossiblefork on Wednesday July 24, 2024 @05:00PM
(#41062054)
So do it locally after predigesting the book, so that you have the entire KV-cache for it.
Then load that KV-cache and add your prompt.