The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by lolinder on Sunday April 09, 2023 @05:15PM (#35506538)

A token per second-ish with a Ryzen 7 5800X. If I run it for too long it gets slower as heat throttling kicks in, I need a better cooling system if I'm going to run it non-stop.

[ Parent | on HN ]

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by morrbo on Sunday April 09, 2023 @05:30PM (#35506681)

i've had the same experience tbh, 7/13/30 on ryzen (local) and intel (server) both on rhel/centos. It's a shame really

[ Parent | on HN ]

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by dandongus on Sunday April 09, 2023 @05:19PM (#35506563)

For a bit of comparison, if you've tested, how fast are 13B or 7B on the same setup?

[ Parent | on HN ]

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by lolinder on Sunday April 09, 2023 @06:31PM (#35507116)

Really fast. I didn't bother timing, but they're faster than ChatGPT by a long shot. I didn't spend very long with them because the quality is so much worse than the 65B.

I should probably go back and try again to see if it's worth it for the extra speed, now that I've played with 65B for a while.

[ Parent | on HN ]

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by ImprobableTruth on Sunday April 09, 2023 @09:59PM (#35508410)

Is this with full or empty context?

[ Parent | on HN ]

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by lolinder on Sunday April 09, 2023 @10:33PM (#35508598)

Good question--I'm counting it from empty and around empty. By the time it gets to full I'm also getting heat throttling (I can tell looking at the temp), so it's hard to know the degree to which the slowdown is one or the other.

[ Parent | on HN ]