News for hackers. Stuff that matters.
Re:Orbit. Mozilla's AI Assistant for Firefox by mhitza on Monday December 30, 2024 @08:56PM (#42555624)
> How does Orbit work?

> Orbit is a Firefox add on that uses AI to summarize and answer queries about web content such as articles and videos.

> When a user asks Orbit to summarize or query content, Orbit gathers the context (eg. text, images, videos, etc.) of the page the user is viewing and provides a summary or answer. Orbit works on websites including Gmail, Wikipedia, NY Times, YouTube, and more.

> For the current version, we are using a Mistral LLM (Mistral 7B) hosted within Mozilla’s GCP instance.

It's interesting they're going with Mistral 7B. Is anybody else using Mistral 7B in production? And in what role?

I've considered using it for general knowledge type questions, and as a way to classify information, but would have never considered it for summarization type tasks due to it's limited context size (8k).

Re:Orbit. Mozilla's AI Assistant for Firefox by pilotneko on Monday December 30, 2024 @08:58PM (#42555633)
Mistral 7B has a context window of 32k. I use it in production for medical summarization tasks supporting appeals and physician advisory services.
Re:Orbit. Mozilla's AI Assistant for Firefox by jiggawatts on Monday December 30, 2024 @09:09PM (#42555679)
Why are you confident to use such a tiny model on something so critical?

I would never use anything smaller than a 70B model for anything even vaguely medical related!

Re:Orbit. Mozilla's AI Assistant for Firefox by pilotneko on Monday December 30, 2024 @10:14PM (#42556071)
I don't use Mistral 7B alone, this is just a component in a RAG-based system. A system that is 1) not clinical facing, 2) not used in clinical decision making, 3) provides in-line references sources for end users to validate information, and 4) is inherently human-in-the-loop.
Re:Orbit. Mozilla's AI Assistant for Firefox by bigstrat2003 on Monday December 30, 2024 @09:41PM (#42555868)
> Why are you confident to use such a tiny model on something so critical?

Don't use any LLM for something critical. They can't be trusted innately due to their design, why on earth would you use them for something where you need reliability?

Re:Orbit. Mozilla's AI Assistant for Firefox by mdaniel on Monday December 30, 2024 @10:37PM (#42556216)
> why on earth would you use them for something where you need reliability?

As very best I can tell the answer is "because bags of cash"

Re:Orbit. Mozilla's AI Assistant for Firefox by add-sub-mul-div on Monday December 30, 2024 @09:26PM (#42555778)
A key thing to remember in this specific moment in history is that the vast majority of people will be as lazy as they can get away with being. People want to leave work and go home and if an LLM lets them do that faster, who cares if it's accurate? It can absorb the blame.
Re:Orbit. Mozilla's AI Assistant for Firefox by N2yhWNXQN3k9 on Monday December 30, 2024 @09:18PM (#42555730)
Its summarisation, who cares if its right as long as you feel confident after reading it? /s.

In my experience, even GPT-4o is terrible at revealing information from things longer than a few pages.

It might be an issue with dimensionality reduction in general though. If you think about it, you can't really take away much of what is contained within any given amount of text with text, unless the source was produced extremely inefficiently.

Producing outlines or maybe a form of abstract, it seems to be okay at, but you would never really know where it fails unless you read the entirety of the source text first to begin with. IMO, its not worth risking unless you plan to read the source anyway or its not really important.

Re:Orbit. Mozilla's AI Assistant for Firefox by someothherguyy on Monday December 30, 2024 @10:14PM (#42556074)
Try to walk through a Wikipedia article having an LLM summarize every few paragraphs, its often wildly inaccurate.
Re:Orbit. Mozilla's AI Assistant for Firefox by HaZeust on Monday December 30, 2024 @09:17PM (#42555722)
Hope your prod is being done currently in this political... Climate.