News for hackers. Stuff that matters.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @04:50PM (#41260218)
I look forward to trying this out, mostly because I’m very frustrated with censored models.

I am experimenting with summarizing and navigating documents for forensic psychiatry work, much of which involves subjects that instantly hit the guard rails of LLMs. So far, I have had zero luck getting help from OpenAI/Anthropic or vendors of their models to request an exception for uncensored models. I need powerful models with good, hipaa-compliant privacy, that won’t balk at topics that have serious effects on people’s lives.

Look, I’m not excited to read hundreds of pages about horrible topics, either. If there were a way to reduce the vicarious trauma of people who do this work without sacrificing accuracy, it would be nice. I’d like to at least experiment. But I’m not going to hold my breath.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by d13 on Thursday August 15, 2024 @05:16PM (#41260457)
All base, “text-completion” models are uncensored, including Llama 3. You can make text-completion models behave like an uncensored “instruct” (chat) model simply by providing it with 10 to 20 examples of a chat dialogue in the initial prompt context, making sure to use the model’s exact prompt format. Once the model notices the pattern, it will continue like that.

Surprisingly few people seem to know this. But, this is how chat models were created in the GPT3/2 era before instruct models became the norm.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by davidzweig on Thursday August 15, 2024 @09:33PM (#41262277)
I was trying this with the original llama model. I guess the model didn't really know it's meant to be a 'knowledgeable ai assistant', but rather simulated chats it had seen. If you asked it, 'how to make brownies', it might reply, 'idk, can't you google it?'.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by d13 on Friday August 16, 2024 @07:10AM (#41265235)
When you prime it with those initial 10-20 examples, the responses need to be in the style that you’d like it to respond to. You can use Claude or ChatGPT to help you write those. The model will then just continue on in that same style.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by viraptor on Friday August 16, 2024 @07:26AM (#41265319)
Unfortunately those examples blow up the cost compared to just asking the question. It's a nice workaround, but not always feasible. (Unless everyone adopts context caching like deepseek and anthropic did)
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by rustcleaner on Thursday August 15, 2024 @11:01PM (#41262748)
I often force compliance out of Llama by starting Llama's response in a very compliant manner.

My favorite use right now is for language translations. I finally feel comfortable browsing foreign languages, knowing Google isn't sitting there as a third party with a coherent view of my foreign language readings!

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by kenmacd on Friday August 16, 2024 @10:53AM (#41266956)
> by providing it with 10 to 20 examples of a chat dialogue in the initial prompt context

Can you recommended examples (or a source of examples) that would have models act similar to the instruct models?

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by torginus on Thursday August 15, 2024 @05:54PM (#41260806)
How good are these models at summarization anyways? I tried uploading obscure books I've already read, to GPT4 and Claude 3 and asked them to summarize the plot and particular details, as well as asking how many times does a particular thing happen in the book, and the results have been hit and miss.

I certainly would not trust these models to create comprehensive and correct summaries of highly sensitive records.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by miki123211 on Thursday August 15, 2024 @07:54PM (#41261741)
For whatever it's worth, asking a model to count is a terrible idea due to how they work.

You may have more luck with a hybrid approach, using LLMs for language understanding and computers for the counting. For example, ask them to write a short, one-line description of every instance where something happens, and then use a traditional program to count the lines.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by simonw on Thursday August 15, 2024 @06:29PM (#41261139)
Asking "how many times does a particular thing happen in the book" is always going to be hard, because LLMs are notoriously bad at counting.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by itake on Thursday August 15, 2024 @07:18PM (#41261525)
I had issues with gpt4 last summer confusing which characters did what and hallucinating characters.

his may of been a context/chunking issue (if that particular section doesn't name the character performing an action), but maybe its better now.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by sweca on Thursday August 15, 2024 @07:26PM (#41261587)
Were you using the API or ChatGPT? ChatGPT uses RAG on large documents which is useless for many tasks.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by awwaiid on Thursday August 15, 2024 @11:33PM (#41262920)
Last summer was like a million years ago! You should try again :)
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by mlyle on Thursday August 15, 2024 @07:00PM (#41261369)
Man, -people- are bad at counting tasks like this.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by campers on Thursday August 15, 2024 @11:31PM (#41262911)
Ask it to output the occurrences as items in a JSON array, then count the array items in code
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by torginus on Friday August 16, 2024 @06:08AM (#41264930)
Not only that, asking an LLM a question like: "List all the occurences where John Doe fights someone in the book. Also list who he fights and why" is a surefire way of getting LLMs to hallucinate, and also give incomplete info.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by kainan-ai on Thursday August 15, 2024 @05:56PM (#41260827)
Yeah its more creative than other fine tunes, you'd need to make a pretty strict system prompt then test before doing anything with sensitive records
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @08:29PM (#41261916)
If you chunk the document well (which is surprisingly tricky with messy documents), then you can do summary via map reduce or other techniques.

To be clear: I don't trust it to provide an accurate summary; it's not meant to replace reading the documents. But it may help to find relevant parts of a document later, or get a reasonable overview of documents before starting a manual review. I expect (but will have to see) if it's better than just opening up a random PDF and starting there.

I recently had a case with over 24k pages of records, some of which were in PDFs that were thousands of pages long. The ability to do semantic search rather than keyword search was useful for when I said to myself, "Didn't I see something about that before? Where was that?"

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by sroussey on Thursday August 15, 2024 @08:44PM (#41262001)
I’ve found summarizing this way is a great idea but the summarizing models are not great themselves. Would even prefer an embedding with the summary as maybe the embedding would be better.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by pnw on Thursday August 15, 2024 @05:00PM (#41260316)
I just tried it and it appears to be censored. "Providing instructions on creating such materials is not advisable for safety and legal reasons."
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by kainan-ai on Thursday August 15, 2024 @05:59PM (#41260867)
If you take the base model and put in a decent system prompt Hermes 3 405b will follow your system prompt instructions pretty well. The one in the discord has a blank system prompt and is just taking the chat as context.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @05:51PM (#41260775)
Well, there goes that idea. The Dolphin ones appear to be the most useful.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by kainan-ai on Thursday August 15, 2024 @06:03PM (#41260896)
Hermes 3 will follow the sys prompt pretty closely if you have a version where you can edit it. In the discord there were a few times it jailbroke pretty aggressively in spite of the blank system prompt.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by smusamashah on Thursday August 15, 2024 @07:44PM (#41261688)
I remember the Salesforce (edit: snowflake arctic instruct actually) LLM being completely uncensored and unaligned. Someone from the team said in response here on HN that they didn't do any alignment whatsoever.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by oidar on Thursday August 15, 2024 @05:36PM (#41260615)
Mistrial-Nemo should be able to do this.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by fsiefken on Thursday August 15, 2024 @06:17PM (#41261049)
Mistral Large 2 is good too, if you've got the memory https://ollama.com/library/mistral-large
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @05:52PM (#41260784)
This is my current go-to. It's not SOTA, but at least it does something.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by oidar on Friday August 16, 2024 @11:16AM (#41267148)
I've found that it's really good at summarizing - as it's context length is huge. Have you noticed any problems with it in that regard?
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by stavros on Thursday August 15, 2024 @04:56PM (#41260290)
Have you tried any abliterated models?
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @05:53PM (#41260796)
Yes, with mixed results.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by ustad on Thursday August 15, 2024 @05:04PM (#41260349)
Any recommendations?
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by chpatrick on Thursday August 15, 2024 @05:13PM (#41260419)
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by reissbaker on Friday August 16, 2024 @02:14AM (#41263600)
Yup, 3.1-70B-Instruct-lorablated is the one I currently recommend too for anti-rejection models — it seems roughly as anti-rejection as the original failspy "abliterated" model, but it works with 128k context since it's based on 3.1 instead of 3 (which only had 8k context). It's currently our second-most popular model on glhf.chat, behind Llama-3.1-405B-Instruct.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by pizza on Thursday August 15, 2024 @05:10PM (#41260399)
failspy’s or mlabonne’s models. Or just look for any model with ‘abliterated’ in the title. Eg try failspy/meta-llama-3-8b-instruct-abliterated-v3 though of course bigger models will probably be better
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by stavros on Thursday August 15, 2024 @05:09PM (#41260380)
No specific ones, but there are some abliteration LoRas for Llama (8B and 70B, I think). Those should be good for what you want.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by rabi_molar on Friday August 16, 2024 @07:26AM (#41265316)
Hadn't heard about the abliteration before, thanks for bringing it up! Here's a HF walkthrough [1] of the concept for anyone else interested in learning more.

[1] https://huggingface.co/blog/mlabonne/abliteration

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by poisson-fish on Thursday August 15, 2024 @05:08PM (#41260374)
try google's gemini models, safety filtering can be completely disabled via cloud studio or api
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by KTibow on Thursday August 15, 2024 @07:16PM (#41261501)
This is the 2nd time I've seen someone confused about this. The safety filtering is a layer on top of the model's built in ability to refuse, similar to OpenAI's "moderation" endpoint.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by og_kalu on Thursday August 15, 2024 @08:53PM (#41262058)
This is true but still Gemini is by far the least restrictive Sota LLM out there (once all safety settings are none)
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by naiv on Thursday August 15, 2024 @05:47PM (#41260732)
Looks like this is only possible with some prior manual action:

To access the BLOCK_NONE setting, you can:

Apply for the allowlist through the Gemini safety filter allowlist form,

or

Switch your account type to monthly invoiced billing with the Google Cloud invoiced billing reference.

Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by phren0logy on Thursday August 15, 2024 @05:54PM (#41260803)
No, because Google still won't explicitly clarify privacy/HIPAA-compliance on these.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by huac on Thursday August 15, 2024 @09:41PM (#41262330)
no. only 4 categories can be blocked, while the others cannot be disabled.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by kainan-ai on Thursday August 15, 2024 @05:55PM (#41260815)
You can try it out right now in the Nous Research discord, its also up on Lambda labs' new chat thing.
Re:Hermes 3: The First Fine-Tuned Llama 3.1 405B Model by azinman2 on Friday August 16, 2024 @02:21AM (#41263636)
How will you know the model summarizes correctly? Getting long context window summarization right isn’t really a solved problem despite what is advertised, and key details can be missed easily but if you don’t read the original docs you won’t know that. This is very troublesome if applied to a legal scenario.